Re: git gc --aggressive led to about 40 times slower "git log --raw"

From: Duy Nguyen <pclouds@gmail.com>
Date: Feb 22, 2014, 13:00 UTC
Message-ID: <CACsJy8AS6FMqMXcsDtUvrC2bgZ90irMXDCh58KjmgQK8+yFwVA@mail.gmail.com>
In-Reply-To: <CACsJy8Cyf6Mu3q1VWH7srCK4m=+UgR4m7RiNkMv-nr8eF4YAJA@mail.gmail.com>

On Sat, Feb 22, 2014 at 4:14 PM, Duy Nguyen <pclouds@gmail.com> wrote:

Show 13 quoted lines

> On Sat, Feb 22, 2014 at 3:53 PM, David Kastrup <dak@gnu.org> wrote:
>> David Kastrup <dak@gnu.org> writes:
>>
>>> Duy Nguyen <pclouds@gmail.com> writes:
>>>
>>>> OK with git://git.savannah.gnu.org/emacs.git we have
>>>>
>>>>  - a 209MB pack with --aggressive
>>>>  - 1.3GB with --depth=50
>>>>  - 1.3GB with --window=4000 --depth=32
>>>>  - 1.3GB with --depth=20
>>>>  - 821MB with --depth=250 for commits --before=2.years.ago, --depth=50
>>>> for the rest

...

>>>>
>>>> I'm not really happy with --depth=250 producing 209MB while
>>>> --depth=250 --before=2.year.ago a 800MB pack. It looks wrong (or maybe
>>>> I did something wrong)

....

Show 6 quoted lines

>> Another thing: did you really use --depth=250 here or did you use
>> --aggressive?  It may be that the latter also sets other options?
>
> I can't use --aggressive because I need to feed revisions directly to
> pack-objects. --aggressive also sets --window=250. Thanks for
> checking. My machine will have another workout session.

And 800MB is reduced to 177MB, containing history older than 2 years. The final pack is 199MB, within the size range of current --aggressive and should be reasonably fast on most operations. Again blame could still hit long delta chains but I think we should just unpack some trees/blobs when we hit long delta chains.

I think we should update --aggressive to do it this way. So

 - gc.aggressiveDepth defaults to 50 (or 20?), this is used for recent history
 - gc.aggressiveDeepDepth defaults to 250 (or smaller??), used for
ancient history
 - gc.aggressiveDeepOption is rev-list a rev-list option to define
"ancient history", default to --before=2.years.ago. This option could
be specified multiple times.

Both packing phases use the same gc.aggressiveWindow. We could add gc.aggressiveDeepWindow too.

GSoC project?

-- 
Duy