Re: rather slow 'git repack' in 'blob:none' partial clones

> On Mon, Apr 05, 2021 at 03:02:33AM +0200, Rafael Silva wrote:
>> > here are trace timings of running 'git gc' in a "normal" and in a
>> > 'blob:none' partial clone:
>> >
>> >   $ git clone --bare https://github.com/git/git git-full.git
>> >   $ GIT_TRACE_PERFORMANCE=2 /usr/bin/time --format='elapsed: %E  max RSS: %Mk' git -C git-full.git/ gc
> [...]
>> >   elapsed: 0:09.51  max RSS: 358964k
>> >
>> >   $ git clone --bare --filter=blob:none https://github.com/git/git git-partial.git
>> >   $ GIT_TRACE_PERFORMANCE=2 /usr/bin/time --format='elapsed: %E  max RSS: %Mk' git -C git-partial.git/ gc
>> >   10:35:47.637735 trace.c:487             performance: 0.000872539 s: git command: /usr/local/libexec/git-core/git pack-refs --all --prune
>> >   10:35:47.675498 trace.c:487             performance: 0.036246403 s: git command: /usr/local/libexec/git-core/git reflog expire --all
>> >   Enumerating objects: 188205, done.
>> >   Counting objects: 100% (188205/188205), done.
>> >   Delta compression using up to 4 threads
>> >   Compressing objects: 100% (66520/66520), done.
>> >   Writing objects: 100% (188205/188205), done.
>> >   Total 188205 (delta 119967), reused 188205 (delta 119967), pack-reused 0
>> >   10:35:50.081709 trace.c:487             performance: 2.402625839 s: git command: /usr/local/libexec/git-core/git pack-objects --local --delta-base-offset objects/pack/.tmp-2946990-pack
>> >   10:35:50.100131 read-cache.c:2315       performance: 0.000009979 s:  read cache ./index
>> >   10:37:04.973541 trace.c:487             performance: 74.885793630 s: git command: /usr/local/libexec/git-core/git pack-objects --local --delta-base-offset objects/pack/.tmp-2946990-pack --keep-true-parents --honor-pack-keep --non-empty --all --reflog --indexed-objects --exclude-promisor-objects --unpack-unreachable=2.weeks.ago
>> >   Removing duplicate objects: 100% (256/256), done.
>> >   10:37:07.482791 trace.c:487             performance: 79.804973525 s: git command: /usr/local/libexec/git-core/git repack -d -l -A --unpack-unreachable=2.weeks.ago
>> >   10:37:07.549333 trace.c:487             performance: 0.008025426 s: git command: /usr/local/libexec/git-core/git prune --expire 2.weeks.ago --exclude-promisor-objects
>> >   10:37:07.552499 trace.c:487             performance: 0.000362981 s: git command: /usr/local/libexec/git-core/git worktree prune --expire 3.months.ago
>> >   10:37:07.554521 trace.c:487             performance: 0.000273834 s: git command: /usr/local/libexec/git-core/git rerere gc
>> >   10:37:10.168233 trace.c:487             performance: 82.533331484 s: git command: git -C git-partial.git/ gc
>> >   elapsed: 1:22.54  max RSS: 1891832k
>> >
>> > Notice the ~9s vs. 82s runtime and ~350M vs. 1.9G memory consumption
>> > increase.  What's going on here?
>> >
>> > Also note that that second 'git pack-objects' invocation doesn't show
>> > any progress for ~75s.
>> >
>> > FWIW, doing the same in a 'tree:0' partial clone is fast.
>> 
>> I'm not expert on the area - by "area": the entire git code base :).
>> However, I was intrigued by this performance numbers and decided to give
>> it a try on the investigation, mostly for learning.
>
> That's the spirit!
>

>> While I'm not sure
>> about the solution of the problem, I decided to share it here with the
>> hope that at least I'll be saving someone else time.
>> 
>> When I was digging into the code and adding trace2_region_*() calls, I
>> notice most of the time spent on the `git gc` (for the reported
>> situation) was in:
>> 
>>        # In builtin/pack-objects.c
>>        static void get_object_list(int ac, const char **av)
>>        {
>>                ...
>>                if (unpack_unreachable)
>>                        loosen_unused_packed_objects();
>>                ...
>>        }
>> 
>> The loosen_unused_packed_objects() will unpack unreachable objects as
>> loose objects, and given that the partial cloned .pack file is
>> incomplete, this result in writing a lot of loose objects in $GIT_DIR
>> increasing the execution time and memory consumption. This can be seen
>> by watching the $GIT_DIR/objects/ during the `git gc` execution on the
>> partial cloned repo.
>
> Indeed, that 'blob:none' partial clone grew in size temporarily to
> over 1.3GB during repacking, but luckily all those unnecessarily
> loosened objects were removed at the end.  I first noticed this issue
> while attempting to repack a considerably larger partial-cloned
> repository, which I aborted because it ate up all the memory...  I
> suppose that even if it didn't use that much memory, it would
> eventually run out of available disk space for all those loose objects
> anyway...
>
>
>> I'm not entirely sure about this (not this late in the day), but it seems to
>> me that we should simply skip the "missing" (promisor) files when
>> operating on a partial clone.
>> 
>> Perhaps something like:
>> 
>> --- >8 ---
>> diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
>> index 525c2d8552..fedf58323d 100644
>> --- a/builtin/pack-objects.c
>> +++ b/builtin/pack-objects.c
>> @@ -3468,6 +3468,8 @@ static int loosened_object_can_be_discarded(const struct object_id *oid,
>>  {
>>         if (!unpack_unreachable_expiration)
>>                 return 0;
>> +       if (exclude_promisor_objects && is_promisor_object(oid))
>> +               return 1;
>>         if (mtime > unpack_unreachable_expiration)
>>                 return 0;
>>         if (oid_array_lookup(&recent_objects, oid) >= 0)
>> --- >8 ---
>> 
>> I'll try to prepare a patch for this change with proper testing, if this
>> turns out to be proper way to handle partial clone repository.
>> 
>> A quick benchmark did show some promising result:
>> 
>>     # built from: 2e36527f23 (The sixth batch, 2021-04-02)
>>     Benchmark #1: ./bin-wrappers/git -C git.git gc
>>           Time (mean ± σ):     135.669 s ±  0.665 s    [User: 42.789 s, System: 91.332 s]
>>           Range (min … max):   134.905 s … 136.115 s    3 runs
>> 
>>     # built from: 2e36527f23 + minor patch (from above)
>>     Benchmark #2: ./bin-wrappers/git -C git.git gc
>>           Time (mean ± σ):     12.586 s ±  0.031 s    [User: 11.462 s, System: 1.365 s]
>>           Range (min … max):   12.553 s … 12.616 s    3 runs
>> 
>>     Summary:
>>           'Benchmark #2' ran 10.78 ± 0.06 times faster than 'Benchmark #1'
>
> I can confirm that you change speeds up repacking considerably and it
> completely eliminates that temporary repo size explision due to
> unpacked objects, but, alas, it doesn't seem to reduce the memory
> usage.
>
> Thanks,
> Gábor