# Git 2.45.2 - memory leaks reports by address sanitizer (asan) 2 messages from 2024-06-10 to 2024-06-11. Participants: Janne Kiiskila, Jeff King. Thread: https://gitlist.dev/t/61616 ## Janne Kiiskila, 2024-06-10 14:37 Subject: Git 2.45.2 - memory leaks reports by address sanitizer (asan) Message-ID: URL: https://gitlist.dev/e/IA0PR14MB6188120169A3C05677F435A585C62%40IA0PR14MB6188.namprd14.prod.outlook.com ``` Hei, Story begins from trying to add weekly builds to one of our jobs in GitHub. It has run earlier just fine, but with something getting updated - it now started failing. The job just hangs when doing a git checkout. The test job was related to address sanitizer (asan) unit tests and turns out that if I do: export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.5" or export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.8" and then try to clone essentially any repo (all I've tried so far) it will hang in GitHub runners. When trying locally - it does not hang, but seems we do have memory leaks according to asan. - locally: Ubuntu 22.04 machine on Intel x68 (Intel Xeon, Lenovo P720). - uname -a: Linux jannek-P720 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May 7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux Are these genuine findings or false reports? These are reporting both with 2.34.1 and also the latest I could get with with the git ppa, 2.54.2. $ ldd /usr/bin/git linux-vdso.so.1 (0x00007fff87d62000) libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007e0abd828000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007e0abd3e4000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e0abd000000) /lib64/ld-linux-x86-64.so.2 (0x00007e0abd8da000) $ git version git version 2.45.2 $ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.8" $ git clone git@github.com:git/sha1collisiondetection.git Cloning into 'sha1collisiondetection'... AddressSanitizer:DEADLYSIGNAL AddressSanitizer:DEADLYSIGNAL AddressSanitizer:DEADLYSIGNAL AddressSanitizer:DEADLYSIGNAL AddressSanitizer:DEADLYSIGNAL AddressSanitizer:DEADLYSIGNAL fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ================================================================= ==1119515==ERROR: LeakSanitizer: detected memory leaks Direct leak of 9 byte(s) in 1 object(s) allocated from: #0 0x7b4d8fc73668 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:439 #1 0x5db29368e7b9 (/usr/bin/git+0x1737b9) Direct leak of 9 byte(s) in 1 object(s) allocated from: #0 0x7b4d8fc73668 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:439 #1 0x5db29368e7b9 (/usr/bin/git+0x1737b9) #2 0x7ffcaa6c5847 ([stack]+0x20847) SUMMARY: AddressSanitizer: 18 byte(s) leaked in 2 allocation(s). You get slightly different ones with asan5. $ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.5" $ git clone git@github.com:git/sha1collisiondetection.git Cloning into 'sha1collisiondetection'... remote: Enumerating objects: 418, done. remote: Total 418 (delta 0), reused 0 (delta 0), pack-reused 418 Receiving objects: 100% (418/418), 497.09 KiB | 1.14 MiB/s, done. Resolving deltas: 100% (244/244), done. ================================================================= ==1115811==ERROR: LeakSanitizer: detected memory leaks Direct leak of 2168 byte(s) in 1 object(s) allocated from: #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153 #1 0x5797e91bb9e7 (/usr/bin/ssh+0x4c9e7) Direct leak of 501 byte(s) in 1 object(s) allocated from: #0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 #1 0x5797e91af4e6 (/usr/bin/ssh+0x404e6) #2 0x6020000167ef () Direct leak of 464 byte(s) in 1 object(s) allocated from: #0 0x7400ef50dc3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163 #1 0x7400eea88114 (/lib/x86_64-linux-gnu/libc.so.6+0x88114) Direct leak of 306 byte(s) in 1 object(s) allocated from: #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153 #1 0x5797e91cd6c7 (/usr/bin/ssh+0x5e6c7) Direct leak of 48 byte(s) in 1 object(s) allocated from: #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153 #1 0x5797e91be54e (/usr/bin/ssh+0x4f54e) Direct leak of 11 byte(s) in 1 object(s) allocated from: #0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 #1 0x5797e91bc5d1 (/usr/bin/ssh+0x4d5d1) Indirect leak of 184 byte(s) in 1 object(s) allocated from: #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153 #1 0x5797e917c58e (/usr/bin/ssh+0xd58e) Indirect leak of 184 byte(s) in 1 object(s) allocated from: #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153 #1 0x5797e917c573 (/usr/bin/ssh+0xd573) Indirect leak of 160 byte(s) in 1 object(s) allocated from: #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153 #1 0x5797e917c532 (/usr/bin/ssh+0xd532) Indirect leak of 51 byte(s) in 5 object(s) allocated from: #0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 #1 0x5797e91bc5d1 (/usr/bin/ssh+0x4d5d1) Indirect leak of 1 byte(s) in 1 object(s) allocated from: #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153 #1 0x5797e91eb7da (/usr/bin/ssh+0x7c7da) SUMMARY: AddressSanitizer: 4078 byte(s) leaked in 15 allocation(s). fatal: remote transport reported error ================================================================= ==1115810==ERROR: LeakSanitizer: detected memory leaks Direct leak of 18 byte(s) in 2 object(s) allocated from: #0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445 #1 0x5afffb42d7b9 (/usr/bin/git+0x1737b9) Direct leak of 9 byte(s) in 1 object(s) allocated from: #0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445 #1 0x5afffb42d7b9 (/usr/bin/git+0x1737b9) #2 0x7fff76e70847 ([stack]+0x20847) Direct leak of 5 byte(s) in 1 object(s) allocated from: #0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445 #1 0x5afffb433040 (/usr/bin/git+0x179040) SUMMARY: AddressSanitizer: 32 byte(s) leaked in 4 allocation(s). Seems to me there's a few potential issues: - GitHub runners hanging (it should not hang, it should exit with error). - Git itself having potential memory leaks. - I am misusing the LD_PRELOAD and it can not or does not as I expect? (i.e. it should just slow down the execution, but not report any failures if there are no leaks/other issues). I have checked the git repo itself and you do have already some address sanitizing test jobs in there so I would assume you do not have leaks. Workaround for now has been to introduce the actual export as late as possible. Best Regards, Janne Kiiskilä ``` ## Jeff King, 2024-06-11 08:14 Subject: Re: Git 2.45.2 - memory leaks reports by address sanitizer (asan) Message-ID: <20240611081414.GH3248245@coredump.intra.peff.net> URL: https://gitlist.dev/e/20240611081414.GH3248245%40coredump.intra.peff.net In-Reply-To: ``` On Mon, Jun 10, 2024 at 02:37:05PM +0000, Janne Kiiskila wrote: > Seems to me there's a few potential issues: > - GitHub runners hanging (it should not hang, it should exit with error). > - Git itself having potential memory leaks. > - I am misusing the LD_PRELOAD and it can not or does not as I expect? > (i.e. it should just slow down the execution, but not report any failures if there are no leaks/other issues). > > I have checked the git repo itself and you do have already some > address sanitizing test jobs in there so I would assume you do not > have leaks. There are most definitely leaks in Git. In our ASan runs we set "detect_leaks=0" in ASAN_OPTIONS. We have a separate leak-checking build, but are working towards having it completely clean. In the meantime, individual scripts in our test suite are marked as leak-free (about half of them). As far as your use of LD_PRELOAD, I'm not sure. Typically you'd compile with "-fsanitize=address", so the compiler is emitting extra code for the sanitizer, in addition to linking against libasan. I don't know enough about ASan to know whether just an LD_PRELOAD is enough to get useful results. But in general it makes sense to me to only set the LD_PRELOAD for the program you are interested in testing. -Peff ```