From: A Large Angry SCM Date: Wed, 06 Sep 2006 23:39:19 GMT Subject: Re: A look at some alternative PACK file encodings Message-ID: <44FF5C27.2040300@gmail.com> In-Reply-To: <9e4733910609061623k73086dbey4a600ecf2852c024@mail.gmail.com> Jon Smirl wrote: > On 9/6/06, A Large Angry SCM wrote: >> TREE objects do not delta or deflate well. > > I can understand why they don't deflate, the path names are pretty > much unique and the sha1s are incompressible. By why don't they delta > well? Does sorting them by size mess up the delta process? My guess would be the TREEs would only delta well against other TREE versions for the same path. > Shawn is doing some prototype work on true dictionary based > compression. I don't know how far along he is but it has potential for > taking 30% off the Mozilla pack. Just looking at the structures in non-BLOBS, I see a lot of potential for the use of a set dictionaries when deflating TREEs and another set of dictionaries when deflating COMMITs and TAGs. The low hanging fruit is to create dictionaries of the most referenced IDs across all TREE or COMMIT/TAG objects.