{"thread":{"id":"61135","subject":"Failures in GitHub Actions linux-leaks and linux-asan-ubsan","startedAt":"2024-03-16T19:20:47Z","lastAt":"2024-03-18T09:08:49Z","messageCount":4,"participants":["Philippe Blain","Junio C Hamano","Linus Arver","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"490795","messageId":"3e217121-f49b-33bd-b76f-df24efca6d14@gmail.com","threadId":"61135","inReplyTo":null,"subject":"Failures in GitHub Actions linux-leaks and linux-asan-ubsan","fromName":"Philippe Blain","fromEmail":"levraiphilippeblain@gmail.com","sentAt":"2024-03-16T19:20:44Z","receivedAt":"2024-03-16T19:20:47Z","isPatch":false,"sender":{"key":"levraiphilippeblain@gmail.com","avatar":"https://avatars.githubusercontent.com/u/44212482?v=4"},"body":"Hi everyone,\n\nYou might have noticed that the linux-leaks and linux-asan-ubsan\njobs in GitHub Actions started failing last week. I investigated \nthis so I'm sharing my findinds in case it helps.\n\nThe failures are due to the new ubuntu-22.04 GitHub Actions image\n(release 20240310.1.0, [1]) which uses a kernel where ASLR is configured\nin a way that is incompatible with ASan and LSan as used in \nthe GCC and Clang versions in that image. More info can be found\nin [2] and [3] and pages linked there.\n\nA workaround was already implemented in the image generation process \n[4], so the next version of the image should work. I think the images\nare released weekly. We could maybe add the same sysctl command to reduce\nthe entropy to our YAML file, or we could live with it for the next week\nor so while waiting for the next image to roll out.\n\nCheers,\n\nPhilippe.\n\n[1] https://github.com/actions/runner-images/commit/d67fa31aeeec3cf0d666d0eb2976b683471c6b90\n[2] https://github.com/actions/runner-images/issues/9491#issuecomment-1989718917\n[3] https://github.com/actions/runner-images/issues/9524#issuecomment-2002065399\n[4] https://github.com/actions/runner-images/pull/9513\n"},{"id":"490807","messageId":"xmqqbk7dn48s.fsf@gitster.g","threadId":"61135","inReplyTo":"3e217121-f49b-33bd-b76f-df24efca6d14@gmail.com","subject":"Re: Failures in GitHub Actions linux-leaks and linux-asan-ubsan","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-16T23:31:15Z","receivedAt":"2024-03-16T23:31:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Philippe Blain writes:\n\n> Hi everyone,\n>\n> You might have noticed that the linux-leaks and linux-asan-ubsan\n> jobs in GitHub Actions started failing last week. I investigated \n> this so I'm sharing my findinds in case it helps.\n>\n> The failures are due to the new ubuntu-22.04 GitHub Actions image\n> (release 20240310.1.0, [1]) which uses a kernel where ASLR is configured\n> in a way that is incompatible with ASan and LSan as used in \n> the GCC and Clang versions in that image. More info can be found\n> in [2] and [3] and pages linked there.\n>\n> A workaround was already implemented in the image generation process \n> [4], so the next version of the image should work. I think the images\n> are released weekly. We could maybe add the same sysctl command to reduce\n> the entropy to our YAML file, or we could live with it for the next week\n> or so while waiting for the next image to roll out.\n\nAh, just what I needed, as I was looking at the CI results and\nnoticed these failed runs. Thanks.\n"},{"id":"490836","messageId":"owlyo7bc2fr9.fsf@fine.c.googlers.com","threadId":"61135","inReplyTo":"3e217121-f49b-33bd-b76f-df24efca6d14@gmail.com","subject":"Re: Failures in GitHub Actions linux-leaks and linux-asan-ubsan","fromName":"Linus Arver","fromEmail":"linusa@google.com","sentAt":"2024-03-18T06:52:42Z","receivedAt":"2024-03-18T06:52:44Z","isPatch":false,"sender":{"key":"linus@ucla.edu","avatar":null},"body":"Philippe Blain writes:\n\n> Hi everyone,\n>\n> You might have noticed that the linux-leaks and linux-asan-ubsan\n> jobs in GitHub Actions started failing last week. I investigated \n> this so I'm sharing my findinds in case it helps.\n\nHuge thanks for looking into this, Philippe!\n\n> The failures are due to the new ubuntu-22.04 GitHub Actions image\n> (release 20240310.1.0, [1]) which uses a kernel where ASLR is configured\n> in a way that is incompatible with ASan and LSan as used in \n> the GCC and Clang versions in that image. More info can be found\n> in [2] and [3] and pages linked there.\n>\n> A workaround was already implemented in the image generation process \n> [4], so the next version of the image should work. I think the images\n> are released weekly. We could maybe add the same sysctl command to reduce\n> the entropy to our YAML file, or we could live with it for the next week\n> or so while waiting for the next image to roll out.\n>\n> Cheers,\n>\n> Philippe.\n>\n> [1] https://github.com/actions/runner-images/commit/d67fa31aeeec3cf0d666d0eb2976b683471c6b90\n> [2] https://github.com/actions/runner-images/issues/9491#issuecomment-1989718917\n> [3] https://github.com/actions/runner-images/issues/9524#issuecomment-2002065399\n> [4] https://github.com/actions/runner-images/pull/9513\n\nI appreciate the links you've documented here for reference (and the\ncare you took to write up [3]). You've got good taste. :)\n"},{"id":"490842","messageId":"20240318090848.GC602575@coredump.intra.peff.net","threadId":"61135","inReplyTo":"3e217121-f49b-33bd-b76f-df24efca6d14@gmail.com","subject":"Re: Failures in GitHub Actions linux-leaks and linux-asan-ubsan","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-03-18T09:08:48Z","receivedAt":"2024-03-18T09:08:49Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Mar 16, 2024 at 03:20:44PM -0400, Philippe Blain wrote:\n\n> The failures are due to the new ubuntu-22.04 GitHub Actions image\n> (release 20240310.1.0, [1]) which uses a kernel where ASLR is configured\n> in a way that is incompatible with ASan and LSan as used in \n> the GCC and Clang versions in that image. More info can be found\n> in [2] and [3] and pages linked there.\n> \n> A workaround was already implemented in the image generation process \n> [4], so the next version of the image should work. I think the images\n> are released weekly. We could maybe add the same sysctl command to reduce\n> the entropy to our YAML file, or we could live with it for the next week\n> or so while waiting for the next image to roll out.\n\nThanks for digging into this! I had done a little but didn't get nearly\nas far. I am happy I can just ignore it and the problem will resolve\nitself. ;)\n\nWhile I have the attention of folks who might be interested in CI\nfailures, let me hijack the thread for a moment: has anybody figured out\nwhy macOS jobs sometimes time out after 6 hours? I assume that is an\nActions limit, and something is just hanging. It sometimes hits\nosx-reftable[0], but sometimes osx-clang[1] and osx-gcc[2]. I've seen it\non my builds and some on git/git (the last one is from git/git).\n\nIt's hard to tell which test is hanging, because the output only shows\nthe finished tests. I tried running them without \"prove\" and doing it\none-by-one, but then the hang doesn't seem to occur (so presumably it's\na race under load). I tried comparing the list of tests reported as\nfinishing versus the total list, but there was nothing enlightening.\nIt's all stuff in t9xxx, which you'd expect to be running near the end\nanyway (and it feels like if there was _one_ test hanging, we should\nfinish everything else, since we run with some parallelism). So it's\nalmost like a bug in \"prove\" or something. But AFAIK it just started\nhappening in the past month or two.\n\n-Peff\n\n[0]: example run: https://github.com/peff/git/actions/runs/8107092556/job/22158038562\n[1]: example run: https://github.com/peff/git/actions/runs/8091601551/job/22110929146\n[2]: example run: https://github.com/git/git/actions/runs/8273234891/job/22636626511\n"}]}