Re: Decompression speed: zip vs lzo

From: Pierre Habouzit <madcoder@debian.org>
Date: Jan 11, 2008, 09:45 UTC
Message-ID: <20080111094516.GD20141@artemis.madism.org>
In-Reply-To: <alpine.LFD.1.00.0801101332150.3054@xanadu.home>

On Thu, Jan 10, 2008 at 08:39:07PM +0000, Nicolas Pitre wrote:

> On Thu, 10 Jan 2008, Pierre Habouzit wrote:

Show 19 quoted lines

> diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
> index a39cb82..252b03e 100644
> --- a/builtin-pack-objects.c
> +++ b/builtin-pack-objects.c
> @@ -433,7 +433,10 @@ static unsigned long write_object(struct sha1file *f,
>  		}
>  		/* compress the data to store and put compressed length in datalen */
>  		memset(&stream, 0, sizeof(stream));
> -		deflateInit(&stream, pack_compression_level);
> +		if (obj_type == OBJ_REF_DELTA || obj_type == OBJ_OFS_DELTA)
> +			deflateInit(&stream, 0);
> +		else
> +			deflateInit(&stream, pack_compression_level);
>  		maxsize = deflateBound(&stream, size);
>  		out = xmalloc(maxsize);
>  		/* Compress it */
> 
> You then only need to run 'git repack -a -f -d' with and without the 
> above patch.

  Using as a PoC a test that is if (size <= 512) instead, I get:

vanilla git:

$ du -k .git/**/*.pack 180808 .git/objects/pack/pack-7bc9f383c92cbffe366da2d2a62b67bb33a53365.pack $ repeat 5 time git blame MAINTAINERS >|/dev/null git blame MAINTAINERS >| /dev/null 7,34s user 0,09s system 99% cpu 7,433 total git blame MAINTAINERS >| /dev/null 7,31s user 0,16s system 100% cpu 7,475 total git blame MAINTAINERS >| /dev/null 7,35s user 0,08s system 100% cpu 7,431 total git blame MAINTAINERS >| /dev/null 7,30s user 0,18s system 99% cpu 7,482 total git blame MAINTAINERS >| /dev/null 7,33s user 0,16s system 99% cpu 7,492 total

With a compression disabled for sizes <= 512:

$ du -k .git/**/*.pack 188840.git/objects/pack/pack-7bc9f383c92cbffe366da2d2a62b67bb33a53365.pack $ repeat 5 time git blame MAINTAINERS >|/dev/null git blame MAINTAINERS >| /dev/null 7,06s user 0,09s system 100% cpu 7,150 total git blame MAINTAINERS >| /dev/null 7,08s user 0,13s system 99% cpu 7,209 total git blame MAINTAINERS >| /dev/null 7,07s user 0,08s system 99% cpu 7,168 total git blame MAINTAINERS >| /dev/null 7,02s user 0,15s system 99% cpu 7,177 total git blame MAINTAINERS >| /dev/null 7,07s user 0,13s system 99% cpu 7,243 total

Okay, the size doesn't even budge, it's not even near being fun. Though we gain 3% of wall clock time

Let's try with a limit of 1024 then !

$ du -k .git/**/*.pack 201725 .git/objects/pack/pack-7bc9f383c92cbffe366da2d2a62b67bb33a53365.pack $ repeat 5 time git blame MAINTAINERS >|/dev/null git blame MAINTAINERS >| /dev/null 6,93s user 0,16s system 77% cpu 9,109 total git blame MAINTAINERS >| /dev/null 6,88s user 0,08s system 99% cpu 6,965 total git blame MAINTAINERS >| /dev/null 6,84s user 0,10s system 99% cpu 6,952 total git blame MAINTAINERS >| /dev/null 6,86s user 0,12s system 99% cpu 6,983 total git blame MAINTAINERS >| /dev/null 6,81s user 0,18s system 99% cpu 6,994 total

Okay, the packs grows 10%, and the blame takes 6% less time.

Okay the numbers are still not that impressive, but my patch doesn't touches _only_ deltas, but also log comments I said, so I've redone my tests with git log and *TADAAAA*:

vanilla git:
    repeat 5 time git log >|/dev/null
    git log >| /dev/null  2,54s user 0,12s system 99% cpu 2,660 total
    git log >| /dev/null  2,52s user 0,12s system 99% cpu 2,653 total
    git log >| /dev/null  2,57s user 0,07s system 99% cpu 2,637 total
    git log >| /dev/null  2,56s user 0,09s system 99% cpu 2,659 total
    git log >| /dev/null  2,54s user 0,10s system 99% cpu 2,660 total

with the 512 octets limit:

    $ repeat 5 time git log >|/dev/null
    git log >| /dev/null  2,10s user 0,10s system 99% cpu 2,193 total
    git log >| /dev/null  2,08s user 0,10s system 99% cpu 2,189 total
    git log >| /dev/null  2,06s user 0,11s system 100% cpu 2,162 total
    git log >| /dev/null  2,04s user 0,13s system 100% cpu 2,172 total
    git log >| /dev/null  2,06s user 0,13s system 99% cpu 2,198 total

    That's already a 20% time reduction.

with the 1024 octets limits:
    $ repeat 5 time git log >|/dev/null
    git log >| /dev/null  1,39s user 0,12s system 99% cpu 1,512 total
    git log >| /dev/null  1,38s user 0,12s system 100% cpu 1,498 total
    git log >| /dev/null  1,41s user 0,10s system 99% cpu 1,514 total
    git log >| /dev/null  1,41s user 0,10s system 100% cpu 1,506 total
    git log >| /dev/null  1,40s user 0,10s system 100% cpu 1,504 total

    Yes that's 43% time reduction !

  As a side note, repacking with the 1024 octets limits takes 4:06 here,
and 4:26 without the limit at all, which is 8% less time. I know it
doesn't matters a lot as repack is a once time operation, but still, it
would speed up git gc --auto which is not something to neglect
completely.

I say it's worth investigating a _lot_, and the patch is that complicated:

diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index a39cb82..f454929 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -433,7 +433,7 @@ static unsigned long write_object(struct sha1file *f,
                }
                /* compress the data to store and put compressed length in datalen */
                memset(&stream, 0, sizeof(stream));
-               deflateInit(&stream, pack_compression_level);
+               deflateInit(&stream, size > 1024 ? pack_compression_level : 0);
                maxsize = deflateBound(&stream, size);
                out = xmalloc(maxsize);
                /* Compress it */

-- 
·O·  Pierre Habouzit
··O                                                madcoder@debian.org
OOO                                                http://www.madism.org