Re: Delta compression not so effective

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mar 1, 2017, 17:36 UTC
Message-ID: <CA+55aFzQ0o2R2kShS=AuKu0TLnfPV-0JCkViqx5J_afCK0Yt5g@mail.gmail.com>
In-Reply-To: <4d2a1852-8c84-2869-78ad-3c863f6dcaf7@gmail.com>

On Wed, Mar 1, 2017 at 5:51 AM, Marius Storm-Olsen <mstormo@gmail.com> wrote:

Show 11 quoted lines

>
> When first importing, I disabled gc to avoid any repacking until completed.
> When done importing, there was 209GB of all loose objects (~670k files).
> With the hopes of quick consolidation, I did a
>     git -c gc.autoDetach=0 -c gc.reflogExpire=0 \
>           -c gc.reflogExpireUnreachable=0 -c gc.rerereresolved=0 \
>           -c gc.rerereunresolved=0 -c gc.pruneExpire=now \
>           gc --prune
> which brought it down to 206GB in a single pack. I then ran
>     git repack -a -d -F --window=350 --depth=250
> which took it down to 203GB, where I'm at right now.

Considering that it was 209GB in loose objects, I don't think it delta-packed the big objects at all.

I wonder if the big objects end up hitting some size limit that causes the delta creation to fail.

For example, we have that HASH_LIMIT that limits how many hashes we'll create for the same hash bucket, because there's some quadratic behavior in the delta algorithm. It triggered with things like big files that have lots of repeated content.

We also have various memory limits, in particular 'window_memory_limit'. That one should default to 0, but maybe you limited it at some point in a config file and forgot about it?

                                     Linus