2.18.0 Regression: packing performance and effectiveness

I had a user report some poor behavior of 'git gc --aggressive' on a certain repo (which I sadly cannot share). Turns out that on this repo, this operation takes about 60% longer and produces a pack roughly twice the expected size.

Naturally, bisecting takes a while but it points to this commit:

0aca34e8269514ebb67676e0470a314615494ae8 is the first bad commit
commit 0aca34e8269514ebb67676e0470a314615494ae8
Author: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Date:   Sat Apr 14 17:35:11 2018 +0200

    pack-objects: shrink delta_size field in struct object_entry

    Allowing a delta size of 64 bits is crazy. Shrink this field down to
    20 bits with one overflow bit.

    If we find an existing delta larger than 1MB, we do not cache
    delta_size at all and will get the value from oe_size(), potentially
    from disk if it's larger than 4GB.

    Note, since DELTA_SIZE() is used in try_delta() code, it must be
    thread-safe. Luckily oe_size() does guarantee this so we it is
    thread-safe.

    Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
    Signed-off-by: Junio C Hamano <gitster@pobox.com>

To put some numbers behind this, I got on a very beefy box (40 cores, 160GB RAM) and ran some comparisons:

Version  Pack (MB)  MaxRSS(kB)  Time (s)
-------  ---------  ----------  --------
 2.17.0     5498     43513628    2494.85
 2.18.0    10531     40449596    4168.94
 fix-v1     5509     42509784    2480.74
 fiv-v2     5509     41644104    2468.25

where fix-v1 and fix-v2 are different patches on top of git-2.18.0 that I'll follow up to this email with. The patches are just meant as discussion starters. I'll add signoffs and proper commit messages if folks actually want those fixes, but I suspect they're just starting points for discussion.