Hi,
I recently tried to enable git maintenance for some repositories on one of my systems.
The services appeared to execute once but then the git processes got stuck the services never exited.
I tried various things to figure out what was going on including strace on the processes. This showed that the git process was constantly calling sched_yield().
I then connected to the process with gdb and took a backtrace:
Show 14 quoted lines
> $ gdb -p 1344308 > (gdb) bt > #0 0x00007fe6058e9beb in sched_yield () from /lib64/libc.so.6 > #1 0x00007fe606312fe5 in __sanitizer::StopTheWorld(void (*)(__sanitizer::SuspendedThreadsList const&, void*), void*) () from /lib64/libasan.so.8 > #2 0x00007fe60632685e in __lsan::LockStuffAndStopTheWorldCallback(dl_phdr_info*, unsigned long, void*) () from /lib64/libasan.so.8 > #3 0x00007fe605950197 in dl_iterate_phdr () from /lib64/libc.so.6 > #4 0x00007fe606326bb0 in __lsan::LockStuffAndStopTheWorld(void (*)(__sanitizer::SuspendedThreadsList const&, void*), __lsan::CheckForLeaksParam*) () > from /lib64/libasan.so.8 > #5 0x00007fe6063235d2 in __lsan::CheckForLeaksOnce() () from /lib64/libasan.so.8 > #6 0x00007fe606323835 in __lsan::DoLeakCheck() () from /lib64/libasan.so.8 > #7 0x00007fe60582943c in __cxa_finalize () from /lib64/libc.so.6 > #8 0x00007fe606205c87 in __do_global_dtors_aux () from /lib64/libasan.so.8 > #9 0x00007ffd86656ba0 in ?? () > #10 0x00007fe606a6d2d2 in _dl_call_fini (closure_map=0x7fe606a620c0) at dl-call_fini.c:43
This shows that we appear to get stuck inside of the leak sanitizer for some reason.
I build git locally and enable the sanitizer, which helps me catch issues both when developing and running git.
I don't really understand why, but some google searching shows that it could be related to some of the way that systemd sets up and isolates the process.
The end result of this setup is that my git maintenance processes run once then never exit, so systemd service never stops, so the timer never restarts git maintenance on the next hour/day/etc.
I was able to work around this by adding an override.conf with
[Service] Environment="ASAN_OPTIONS=detect_leaks=0"
Which disables the asan leak detection, and prevents this particular issue.
I thought I'd post this so that anyone else running into this issue might hopefully find the fix or workaround instead of wondering why git was getting stuck doing nothing for days on end.