{"thread":{"id":"61616","subject":"Git 2.45.2 - memory leaks reports by address sanitizer (asan)","startedAt":"2024-06-10T14:37:10Z","lastAt":"2024-06-11T08:14:16Z","messageCount":2,"participants":["Janne Kiiskila","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"496775","messageId":"IA0PR14MB6188120169A3C05677F435A585C62@IA0PR14MB6188.namprd14.prod.outlook.com","threadId":"61616","inReplyTo":null,"subject":"Git 2.45.2 - memory leaks reports by address sanitizer (asan)","fromName":"Janne Kiiskila","fromEmail":"janne.kiiskila@izumanetworks.com","sentAt":"2024-06-10T14:37:05Z","receivedAt":"2024-06-10T14:37:10Z","isPatch":false,"sender":{"key":"janne.kiiskila@izumanetworks.com","avatar":null},"body":"Hei,\n\nStory begins from trying to add weekly builds to one of our jobs in GitHub. It has run earlier just fine, but with something getting updated - it now started failing. The job just hangs when doing a git checkout. The test job was related to address sanitizer (asan) unit tests and turns out that if I do:\n\nexport LD_PRELOAD=\"/usr/lib/x86_64-linux-gnu/libasan.so.5\"\nor\nexport LD_PRELOAD=\"/usr/lib/x86_64-linux-gnu/libasan.so.8\"\n\nand then try to clone essentially any repo (all I've tried so far) it will hang in GitHub runners.\n\nWhen trying locally - it does not hang, but seems we do have memory leaks according to asan.\n- locally: Ubuntu 22.04 machine on Intel x68 (Intel Xeon, Lenovo P720).\n- uname -a: Linux jannek-P720 6.5.0-35-generic #35~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue May  7 09:00:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux\n\nAre these genuine findings or false reports? These are reporting both with 2.34.1 and also the latest I could get with with the git ppa, 2.54.2.\n\n$ ldd /usr/bin/git\n\tlinux-vdso.so.1 (0x00007fff87d62000)\n\tlibpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007e0abd828000)\n\tlibz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007e0abd3e4000)\n\tlibc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e0abd000000)\n\t/lib64/ld-linux-x86-64.so.2 (0x00007e0abd8da000)\n$ git version\ngit version 2.45.2\n$ export LD_PRELOAD=\"/usr/lib/x86_64-linux-gnu/libasan.so.8\"\n$ git clone git@github.com:git/sha1collisiondetection.git\nCloning into 'sha1collisiondetection'...\nAddressSanitizer:DEADLYSIGNAL\nAddressSanitizer:DEADLYSIGNAL\nAddressSanitizer:DEADLYSIGNAL\nAddressSanitizer:DEADLYSIGNAL\nAddressSanitizer:DEADLYSIGNAL\nAddressSanitizer:DEADLYSIGNAL\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n\n=================================================================\n==1119515==ERROR: LeakSanitizer: detected memory leaks\n\nDirect leak of 9 byte(s) in 1 object(s) allocated from:\n    #0 0x7b4d8fc73668 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:439\n    #1 0x5db29368e7b9  (/usr/bin/git+0x1737b9)\n\nDirect leak of 9 byte(s) in 1 object(s) allocated from:\n    #0 0x7b4d8fc73668 in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cpp:439\n    #1 0x5db29368e7b9  (/usr/bin/git+0x1737b9)\n    #2 0x7ffcaa6c5847  ([stack]+0x20847)\n\nSUMMARY: AddressSanitizer: 18 byte(s) leaked in 2 allocation(s).\n\nYou get slightly different ones with asan5.\n\n\n$ export LD_PRELOAD=\"/usr/lib/x86_64-linux-gnu/libasan.so.5\"\n$ git clone git@github.com:git/sha1collisiondetection.git\nCloning into 'sha1collisiondetection'...\nremote: Enumerating objects: 418, done.\nremote: Total 418 (delta 0), reused 0 (delta 0), pack-reused 418\nReceiving objects: 100% (418/418), 497.09 KiB | 1.14 MiB/s, done.\nResolving deltas: 100% (244/244), done.\n\n=================================================================\n==1115811==ERROR: LeakSanitizer: detected memory leaks\n\nDirect leak of 2168 byte(s) in 1 object(s) allocated from:\n    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153\n    #1 0x5797e91bb9e7  (/usr/bin/ssh+0x4c9e7)\n\nDirect leak of 501 byte(s) in 1 object(s) allocated from:\n    #0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144\n    #1 0x5797e91af4e6  (/usr/bin/ssh+0x404e6)\n    #2 0x6020000167ef  (<unknown module>)\n\nDirect leak of 464 byte(s) in 1 object(s) allocated from:\n    #0 0x7400ef50dc3e in __interceptor_realloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:163\n    #1 0x7400eea88114  (/lib/x86_64-linux-gnu/libc.so.6+0x88114)\n\nDirect leak of 306 byte(s) in 1 object(s) allocated from:\n    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153\n    #1 0x5797e91cd6c7  (/usr/bin/ssh+0x5e6c7)\n\nDirect leak of 48 byte(s) in 1 object(s) allocated from:\n    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153\n    #1 0x5797e91be54e  (/usr/bin/ssh+0x4f54e)\n\nDirect leak of 11 byte(s) in 1 object(s) allocated from:\n    #0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144\n    #1 0x5797e91bc5d1  (/usr/bin/ssh+0x4d5d1)\n\nIndirect leak of 184 byte(s) in 1 object(s) allocated from:\n    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153\n    #1 0x5797e917c58e  (/usr/bin/ssh+0xd58e)\n\nIndirect leak of 184 byte(s) in 1 object(s) allocated from:\n    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153\n    #1 0x5797e917c573  (/usr/bin/ssh+0xd573)\n\nIndirect leak of 160 byte(s) in 1 object(s) allocated from:\n    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153\n    #1 0x5797e917c532  (/usr/bin/ssh+0xd532)\n\nIndirect leak of 51 byte(s) in 5 object(s) allocated from:\n    #0 0x7400ef50d808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144\n    #1 0x5797e91bc5d1  (/usr/bin/ssh+0x4d5d1)\n\nIndirect leak of 1 byte(s) in 1 object(s) allocated from:\n    #0 0x7400ef50da06 in __interceptor_calloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:153\n    #1 0x5797e91eb7da  (/usr/bin/ssh+0x7c7da)\n\nSUMMARY: AddressSanitizer: 4078 byte(s) leaked in 15 allocation(s).\nfatal: remote transport reported error\n\n=================================================================\n==1115810==ERROR: LeakSanitizer: detected memory leaks\n\nDirect leak of 18 byte(s) in 2 object(s) allocated from:\n    #0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445\n    #1 0x5afffb42d7b9  (/usr/bin/git+0x1737b9)\n\nDirect leak of 9 byte(s) in 1 object(s) allocated from:\n    #0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445\n    #1 0x5afffb42d7b9  (/usr/bin/git+0x1737b9)\n    #2 0x7fff76e70847  ([stack]+0x20847)\n\nDirect leak of 5 byte(s) in 1 object(s) allocated from:\n    #0 0x7dcc516963ed in __interceptor_strdup ../../../../src/libsanitizer/asan/asan_interceptors.cc:445\n    #1 0x5afffb433040  (/usr/bin/git+0x179040)\n\nSUMMARY: AddressSanitizer: 32 byte(s) leaked in 4 allocation(s).\n\nSeems to me there's a few potential issues:\n- GitHub runners hanging (it should not hang, it should exit with error).\n- Git itself having potential memory leaks.\n- I am misusing the LD_PRELOAD and it can not or does not as I expect?\n  (i.e. it should just slow down the execution, but not report any failures if there are no leaks/other issues).\n\nI have checked the git repo itself and you do have already some address sanitizing test jobs in there so I would assume you do not have leaks. \n\nWorkaround for now has been to introduce the actual export as late as possible.\n\nBest Regards,\n\n\nJanne Kiiskilä\n"},{"id":"496842","messageId":"20240611081414.GH3248245@coredump.intra.peff.net","threadId":"61616","inReplyTo":"IA0PR14MB6188120169A3C05677F435A585C62@IA0PR14MB6188.namprd14.prod.outlook.com","subject":"Re: Git 2.45.2 - memory leaks reports by address sanitizer (asan)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-06-11T08:14:14Z","receivedAt":"2024-06-11T08:14:16Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 10, 2024 at 02:37:05PM +0000, Janne Kiiskila wrote:\n\n> Seems to me there's a few potential issues:\n> - GitHub runners hanging (it should not hang, it should exit with error).\n> - Git itself having potential memory leaks.\n> - I am misusing the LD_PRELOAD and it can not or does not as I expect?\n>   (i.e. it should just slow down the execution, but not report any failures if there are no leaks/other issues).\n> \n> I have checked the git repo itself and you do have already some\n> address sanitizing test jobs in there so I would assume you do not\n> have leaks. \n\nThere are most definitely leaks in Git. In our ASan runs we set\n\"detect_leaks=0\" in ASAN_OPTIONS. We have a separate leak-checking\nbuild, but are working towards having it completely clean. In the\nmeantime, individual scripts in our test suite are marked as leak-free\n(about half of them).\n\nAs far as your use of LD_PRELOAD, I'm not sure. Typically you'd compile\nwith \"-fsanitize=address\", so the compiler is emitting extra code for\nthe sanitizer, in addition to linking against libasan. I don't know\nenough about ASan to know whether just an LD_PRELOAD is enough to get\nuseful results. But in general it makes sense to me to only set the\nLD_PRELOAD for the program you are interested in testing.\n\n-Peff\n"}]}