Re: Decompression speed: zip vs lzo
On Thu, Jan 10, 2008 at 09:30:59PM +0000, Nicolas Pitre wrote:
Show 22 quoted lines
> On Thu, 10 Jan 2008, Linus Torvalds wrote:
>
> >
> >
> > On Thu, 10 Jan 2008, Nicolas Pitre wrote:
> > >
> > > Here's my rather surprising results:
> > >
> > > My kernel repo pack size without the patch: 184275401 bytes
> > > Same repo with the above patch applied: 205204930 bytes
> > >
> > > So it is only 11% larger. I was expecting much more.
> >
> > It's probably worth doing those statistics on some other projects.
> >
> > Maybe the difference to other repositories isn't huge, and maybe the
> > kernel *is* a good test-case, but I just wouldn't take that for granted.
>
> Obviously.
>
> This was a really crud test, and my initial goal was to quickly dismiss
> Pierre's assertion. Turns out that he wasn't that wrong after all,
Well that wasn't a random assertion, I made it, because I assumed that
a delta is usually less than a few hundred bytes, and as compression is
applied only to the delta without context, you end up packing 500 bytes
per 500 bytes which will seldomly have excellent compression ratios.
Show 5 quoted lines
> and
> if a significant increase in access speed by avoiding zlib for 82% of
> object accesses can also be demonstrated for the kernel, then we have an
> opportunity for some optimization tradeoff with no backward
> compatibility concerns.
Well, one could use the fact that deltas are not packed to avoid
copying them around, and that will _necessarily_ become a gain (you can
read them where they have been mmapped for instance). The number that
were given for git annotate use a compression of `0' which doesn't use
that fact, and I wouldn't be surprised to see a noticeable gain if one
does that.
And actually, maybe that it's not the deltas we should not pack, but
objects under a certain size (say 512 bytes e.g. ?), whichever type they
have, and to have the code exploit that fact for real, and avoid copies.
With this criterion, I expect the repository to not grow a lot larger
(I'd say quite less than the 10% you had, as even in the kernel, there
_are_ some larger deltas, and we definitely loose space for them, I'd
expect less than a 5% size variation), and I _think_ it's worth
investigating. At least I expect visible results on commands (like blame
of even log[0]) that go through a lot of small objects to see 10 to 20%
increase speed (backed up by some experience I have in avoiding copies
in not-so-similar cases though, so it may be less, and I'll stand
corrected -- and disappointed, a bit).
[0] If I'm correct commit messages are "objects" on their own, and I
don't expect them to be very often over 512 octets.--
·O· Pierre Habouzit
··O madcoder@debian.org
OOO http://www.madism.org
Pierre Habouzit <madcoder@debian.org>