Re: Compression and dictionaries

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Aug 14, 2006, 19:38 UTC
Message-ID: <Pine.LNX.4.63.0608142133150.28360@wbgn013.biozentrum.uni-wuerzburg.de>
In-Reply-To: <Pine.LNX.4.63.0608141206260.14796@qynat.qvtvafvgr.pbz>

Hi,

On Mon, 14 Aug 2006, David Lang wrote:

Show 11 quoted lines

> On Mon, 14 Aug 2006, Jon Smirl wrote:
> 
> > Bit errors can mess the pack up in lots of ways. If it hits a commit 
> > you won't be able to follow the tree back in time. Packs were never 
> > designed to be error tolerant.
> 
> I'm not claiming that this is a problem, I'm reponding to other people's 
> claim that useing a global dictionary for a pack is a problem becouse if 
> something happens to that dictionary the whole pack is worthless by 
> pointing out that, if this is viewed as a real problem, it's easy to 
> solve.

Let's not solve problems we do not have. I refuse to think about this problem further, before there are actually some hard numbers showing an improvement there. If the numbers do not show an improvement, we do not have that problem at all!

Make a _global_ dictionary, optimize the heck out of it, _use_ that dictionary to repack a sizeable repository (I think linux-2.6.git should be enough for first tests), and tell the world about the size of the dictionary, the original pack, and the new pack.

You would not need to implement this cleanly, just a hack to prove that this idea is worth following up.

Ciao, Dscho