threads / discuss / 61616

Git 2.45.2 - memory leaks reports by address sanitizer (asan)

Subject: Git 2.45.2 - memory leaks reports by address sanitizer (asan)

## tl;dr

2 messages between Jun 10, 2024 and Jun 11, 2024.

replies: 1people: 2as markdown or json

Janne Kiiskila· Jun 10, 2024, 14:37 UTC · lore
Hei,
Story begins from trying to add weekly builds to one of our jobs in GitHub. It has run earlier just fine, but with something getting updated - it now started failing. The job just hangs when doing a git checkout. The test job was related to address sanitizer (asan) unit tests and turns out that if I do:

export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.5" or export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.8"

and then try to clone essentially any repo (all I've tried so far) it will hang in GitHub runners.
When trying locally - it does not hang, but seems we do have memory leaks according to asan.
- locally: Ubuntu 22.04 machine on Intel x68 (Intel Xeon, Lenovo P720).
- uname -a: Linux jannek-P720 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May  7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Are these genuine findings or false reports? These are reporting both with 2.34.1 and also the latest I could get with with the git ppa, 2.54.2.
$ ldd /usr/bin/git
	linux-vdso.so.1 (0x00007fff87d62000)
	libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007e0abd828000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007e0abd3e4000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e0abd000000)
	/lib64/ld-linux-x86-64.so.2 (0x00007e0abd8da000)
$ git version
git version 2.45.2
$ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.8"
$ git clone git@github.com:git/sha1collisiondetection.git
Cloning into 'sha1collisiondetection'...
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

================================================================= ==1119515==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 9 byte(s) in 1 object(s) allocated from:
    #0 0x7b4d8fc73668 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:439
    #1 0x5db29368e7b9  (/usr/bin/git+0x1737b9)
Direct leak of 9 byte(s) in 1 object(s) allocated from:
    #0 0x7b4d8fc73668 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:439
    #1 0x5db29368e7b9  (/usr/bin/git+0x1737b9)
    #2 0x7ffcaa6c5847  ([stack]+0x20847)
SUMMARY: AddressSanitizer: 18 byte(s) leaked in 2 allocation(s).
You get slightly different ones with asan5.

$ export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libasan.so.5" $ git clone git@github.com:git/sha1collisiondetection.git Cloning into 'sha1collisiondetection'... remote: Enumerating objects: 418, done. remote: Total 418 (delta 0), reused 0 (delta 0), pack-reused 418 Receiving objects: 100% (418/418), 497.09 KiB | 1.14 MiB/s, done. Resolving deltas: 100% (244/244), done.

================================================================= ==1115811==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 2168 byte(s) in 1 object(s) allocated from:
    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
    #1 0x5797e91bb9e7  (/usr/bin/ssh+0x4c9e7)
Direct leak of 501 byte(s) in 1 object(s) allocated from:
    #0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5797e91af4e6  (/usr/bin/ssh+0x404e6)
    #2 0x6020000167ef  (<unknown module>)
Direct leak of 464 byte(s) in 1 object(s) allocated from:
    #0 0x7400ef50dc3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163
    #1 0x7400eea88114  (/lib/x86_64-linux-gnu/libc.so.6+0x88114)
Direct leak of 306 byte(s) in 1 object(s) allocated from:
    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
    #1 0x5797e91cd6c7  (/usr/bin/ssh+0x5e6c7)
Direct leak of 48 byte(s) in 1 object(s) allocated from:
    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
    #1 0x5797e91be54e  (/usr/bin/ssh+0x4f54e)
Direct leak of 11 byte(s) in 1 object(s) allocated from:
    #0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5797e91bc5d1  (/usr/bin/ssh+0x4d5d1)
Indirect leak of 184 byte(s) in 1 object(s) allocated from:
    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
    #1 0x5797e917c58e  (/usr/bin/ssh+0xd58e)
Indirect leak of 184 byte(s) in 1 object(s) allocated from:
    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
    #1 0x5797e917c573  (/usr/bin/ssh+0xd573)
Indirect leak of 160 byte(s) in 1 object(s) allocated from:
    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
    #1 0x5797e917c532  (/usr/bin/ssh+0xd532)
Indirect leak of 51 byte(s) in 5 object(s) allocated from:
    #0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x5797e91bc5d1  (/usr/bin/ssh+0x4d5d1)
Indirect leak of 1 byte(s) in 1 object(s) allocated from:
    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153
    #1 0x5797e91eb7da  (/usr/bin/ssh+0x7c7da)
SUMMARY: AddressSanitizer: 4078 byte(s) leaked in 15 allocation(s).
fatal: remote transport reported error

================================================================= ==1115810==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 18 byte(s) in 2 object(s) allocated from:
    #0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445
    #1 0x5afffb42d7b9  (/usr/bin/git+0x1737b9)
Direct leak of 9 byte(s) in 1 object(s) allocated from:
    #0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445
    #1 0x5afffb42d7b9  (/usr/bin/git+0x1737b9)
    #2 0x7fff76e70847  ([stack]+0x20847)
Direct leak of 5 byte(s) in 1 object(s) allocated from:
    #0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445
    #1 0x5afffb433040  (/usr/bin/git+0x179040)
SUMMARY: AddressSanitizer: 32 byte(s) leaked in 4 allocation(s).
Seems to me there's a few potential issues:
- GitHub runners hanging (it should not hang, it should exit with error).
- Git itself having potential memory leaks.
- I am misusing the LD_PRELOAD and it can not or does not as I expect?
  (i.e. it should just slow down the execution, but not report any failures if there are no leaks/other issues).
I have checked the git repo itself and you do have already some address sanitizing test jobs in there so I would assume you do not have leaks. 
Workaround for now has been to introduce the actual export as late as possible.
Best Regards,
Janne Kiiskilä
Jeff King· Jun 11, 2024, 08:14 UTC · re: Janne Kiiskila · lore

Re: Git 2.45.2 - memory leaks reports by address sanitizer (asan)

On Mon, Jun 10, 2024 at 02:37:05PM +0000, Janne Kiiskila wrote:
Show 9 quoted lines
> Seems to me there's a few potential issues:
> - GitHub runners hanging (it should not hang, it should exit with error).
> - Git itself having potential memory leaks.
> - I am misusing the LD_PRELOAD and it can not or does not as I expect?
>   (i.e. it should just slow down the execution, but not report any failures if there are no leaks/other issues).
> 
> I have checked the git repo itself and you do have already some
> address sanitizing test jobs in there so I would assume you do not
> have leaks.

There are most definitely leaks in Git. In our ASan runs we set "detect_leaks=0" in ASAN_OPTIONS. We have a separate leak-checking build, but are working towards having it completely clean. In the meantime, individual scripts in our test suite are marked as leak-free (about half of them).

As far as your use of LD_PRELOAD, I'm not sure. Typically you'd compile with "-fsanitize=address", so the compiler is emitting extra code for the sanitizer, in addition to linking against libasan. I don't know enough about ASan to know whether just an LD_PRELOAD is enough to get useful results. But in general it makes sense to me to only set the LD_PRELOAD for the program you are interested in testing.

-Peff

← back to recent threads