Re: Decompression speed: zip vs lzo

From: Marco Costalba <mcostalba@gmail.com>
Date: Jan 10, 2008, 11:45 UTC
Message-ID: <e5bfff550801100345i20cb3030mf04a11d610fda6f7@mail.gmail.com>
In-Reply-To: <e5bfff550801092255wc852252m9086567a88b1ae99@mail.gmail.com>

On Jan 10, 2008 7:55 AM, Marco Costalba <mcostalba@gmail.com> wrote:

Show 13 quoted lines

>
> [1] where inflate() is called:
>
> -inflate_it() in builtin-apply.c
> -check_pack_inflate() in builtin-pack-objects.c
> -get_data() in builtin-unpack-objects.c
> -fwrite_sha1_file() in http-push.c and http-walker.c  [mmm interesting
> same function in two files, also the signature and the contents seems
> the same....]
> -unpack_entry_data() in index-pack.c
> -unpack_sha1_header(), unpack_sha1_rest(), get_size_from_delta(),
> unpack_compressed_entry, write_sha1_from_fd() in sha1_file.c
>

Looking at the git sources I have found that zip routines are candidate for a cleaning up, as example the more or less very similar lines of code are repeated many times in git files:

memset(&stream, 0, sizeof(stream)); deflateInit(&stream, pack_compression_level); maxsize = deflateBound(&stream, size); out = xmalloc(maxsize); stream.next_out = out; stream.avail_out = maxsize;

So what I'm planning to do to test with different algorithms is first a cleanup work that is more or less the following

- Remove #include <zlib.h> from cache.h and substitute with #include
"compress.h"

- Add #include <zlib.h> where it is "really" intended as example archive-zip.c

- Rename inflate()/deflate() and other zlib calls with corresponding
  zlib_inflate()
  zlib_deflate()

and declared in compress.h

- Define zlib_inflate() and friends as simple wrappers to
corresponding zlib function

- Test if everything is ok (should be only code shuffling/renaming until now)

- Start cleaning up as example adding a do_deflateInit() that wraps
all the code I have reported above and that involves deflateInit()

- When compression routines are cleaned up add new functions

do_inflate(), do_deflate() instead of zlib_* ones that wrap the compression alghorithm dispatching logic.

Dispatching could be choose in different ways going from

- compile time (at #define level)
- config (some configuration value stored in some global variable)
- dynamic (at run time, with no configuration needed, I have some
ideas on this ;-)

Comments?

Thanks Marco