{"thread":{"id":"63301","subject":"[RFH] fedora: pedantic fails complaining that awk is missing","startedAt":"2025-04-15T23:58:19Z","lastAt":"2025-04-16T14:03:12Z","messageCount":4,"participants":["Junio C Hamano","Todd Zullinger","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"516240","messageId":"xmqq34e9kmef.fsf@gitster.g","threadId":"63301","inReplyTo":null,"subject":"[RFH] fedora: pedantic fails complaining that awk is missing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-15T23:58:16Z","receivedAt":"2025-04-15T23:58:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"GitHub Actions CI started failing the pedantic (fedora) job at the\ntip of 'master' few days ago, and the log claims the failure is due\nto missing \"awk\". Even though we have seen a few topics to rewrite\nPerl scriptlet, and I think at least one of them uses awk, but they\nhaven't hit 'master' yet, so it is puzzling why this started failing\nall of a sudden.\n\nSo here is a band-aid. I wouldn't be surprised if the base image\nwas updated without telling us. We have seen Ubuntu 20.04 base\nimage retired to cause CI failures as well.\n\nAlmalinux does not even seem to have awk available as a package\n(or it may be called differently, but we are not suffering from\nthe lack of awk on that platform anyway), so make sure we ask for\nawk only on fedora.\n\nSigned-off-by: Junio C Hamano \n---\n\n * As the scheduled retirement of U20.04 image broke CI jobs running\n for 'master', I started looking at making it pass again with the\n minimum change, i.e. by merging dd/sparse-glibc-workaround and\n js/ci-github-update-ubuntu topics.\n\n Then I somehow found that another job is broken. And this patch\n seems to make it work. Not knowing how it got broken is\n unsatisfactory, though.\n\n ci/install-dependencies.sh | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 8700c0f292..a7c613ce4c 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -30,8 +30,10 @@ alpine-*)\n \t\tbash cvs gnupg perl-cgi perl-dbd-sqlite perl-io-tty >/dev/null\n \t;;\n fedora-*|almalinux-*)\n+\tawk=\n+\tcase \"$distro\" in fedora-*) awk=awk ;; esac\n \tdnf -yq update >/dev/null &&\n-\tdnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\tdnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel $awk >/dev/null\n \t;;\n ubuntu-*|i386/ubuntu-*|debian-*)\n \t# Required so that apt doesn't wait for user input on certain packages.\n-- \n2.49.0-511-g173acbf1d8\n\n"},{"id":"516247","messageId":"Z_80oMscHcR4pyvf@teonanacatl.net","threadId":"63301","inReplyTo":"xmqq34e9kmef.fsf@gitster.g","subject":"Re: [RFH] fedora: pedantic fails complaining that awk is missing","fromName":"Todd Zullinger","fromEmail":"tmz@pobox.com","sentAt":"2025-04-16T04:40:00Z","receivedAt":"2025-04-16T04:40:03Z","isPatch":false,"sender":{"key":"tmz@pobox.com","avatar":"https://avatars.githubusercontent.com/u/806319?v=4"},"body":"Hi,\n\nJunio C Hamano wrote:\n> GitHub Actions CI started failing the pedantic (fedora) job at the\n> tip of 'master' few days ago, and the log claims the failure is due\n> to missing \"awk\". Even though we have seen a few topics to rewrite\n> Perl scriptlet, and I think at least one of them uses awk, but they\n> haven't hit 'master' yet, so it is puzzling why this started failing\n> all of a sudden.\n> \n> So here is a band-aid. I wouldn't be surprised if the base image\n> was updated without telling us. We have seen Ubuntu 20.04 base\n> image retired to cause CI failures as well.\n> \n> Almalinux does not even seem to have awk available as a package\n> (or it may be called differently, but we are not suffering from\n> the lack of awk on that platform anyway), so make sure we ask for\n> awk only on fedora.\n> \n> Signed-off-by: Junio C Hamano \n> ---\n> \n> * As the scheduled retirement of U20.04 image broke CI jobs running\n> for 'master', I started looking at making it pass again with the\n> minimum change, i.e. by merging dd/sparse-glibc-workaround and\n> js/ci-github-update-ubuntu topics.\n> \n> Then I somehow found that another job is broken. And this patch\n> seems to make it work. Not knowing how it got broken is\n> unsatisfactory, though.\n\nI can hazard a guess (or several)...\n\nIt is likely that the image pointed to by the fedora:latest\ntag has moved from fedora 41 to 42, which was released\ntoday. The fedora 41 container images have awk installed\nwhile the fedora 42 images do not. That change is, I\nsuspect, just part of reducing the size of the base\ncontainer images.\n\nIn both AlmaLinux and Fedora (as well as other RHEL\nderivatives/relatives), awk is provided by the gawk package.\n\nOn Fedora, `dnf install awk` uses the package filelist data\nto determine that /usr/bin/awk is provided by gawk and\ninstalls gawk as a result.\n\nOn AlmaLinux (8 & 9, by my quick testing), that is not the\ncase and you'd need to use `dnf install gawk` or `dnf\ninstall '*bin/awk'` to get it installed. Though awk is\nincluded in the current AlmaLinux 8 and 9 images, so it\nisn't strictly needed. But it's probably better to be\nexplicit that we need it installed, as a defense against\nsome future change to the AlmaLinux container removing awk.\n\nUsing gawk (or even '*bin/awk') would likely be clearer.\nThat avoids relying on 1) a case statement to set an awk\nvariable; and 2) the different behavior of older and newer\nreleases of dnf with respect to the install argument\nmatching on the names of binaries provided by the package.\n\nBut that's certainly subjective. ;)\n\n> ci/install-dependencies.sh | 4 +++-\n> 1 file changed, 3 insertions(+), 1 deletion(-)\n> \n> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> index 8700c0f292..a7c613ce4c 100755\n> --- a/ci/install-dependencies.sh\n> +++ b/ci/install-dependencies.sh\n> @@ -30,8 +30,10 @@ alpine-*)\n> \t\tbash cvs gnupg perl-cgi perl-dbd-sqlite perl-io-tty >/dev/null\n> \t;;\n> fedora-*|almalinux-*)\n> +\tawk=\n> +\tcase \"$distro\" in fedora-*) awk=awk ;; esac\n> \tdnf -yq update >/dev/null &&\n> -\tdnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n> +\tdnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel $awk >/dev/null\n> \t;;\n> ubuntu-*|i386/ubuntu-*|debian-*)\n> \t# Required so that apt doesn't wait for user input on certain packages.\n\n-- \nTodd\n"},{"id":"516249","messageId":"81209cfd-049e-c853-9c4a-2cf9803e56a4@gmx.de","threadId":"63301","inReplyTo":"Z_80oMscHcR4pyvf@teonanacatl.net","subject":"Re: [RFH] fedora: pedantic fails complaining that awk is missing","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2025-04-16T05:31:01Z","receivedAt":"2025-04-16T05:31:02Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Todd,\n\nOn Wed, 16 Apr 2025, Todd Zullinger wrote:\n\n> Junio C Hamano wrote:\n> > GitHub Actions CI started failing the pedantic (fedora) job at the\n> > tip of 'master' few days ago, and the log claims the failure is due\n> > to missing \"awk\". Even though we have seen a few topics to rewrite\n> > Perl scriptlet, and I think at least one of them uses awk, but they\n> > haven't hit 'master' yet, so it is puzzling why this started failing\n> > all of a sudden.\n> > \n> > So here is a band-aid. I wouldn't be surprised if the base image\n> > was updated without telling us. We have seen Ubuntu 20.04 base\n> > image retired to cause CI failures as well.\n> > \n> > Almalinux does not even seem to have awk available as a package\n> > (or it may be called differently, but we are not suffering from\n> > the lack of awk on that platform anyway), so make sure we ask for\n> > awk only on fedora.\n> > \n> > Signed-off-by: Junio C Hamano \n> > ---\n> > \n> > * As the scheduled retirement of U20.04 image broke CI jobs running\n> > for 'master', I started looking at making it pass again with the\n> > minimum change, i.e. by merging dd/sparse-glibc-workaround and\n> > js/ci-github-update-ubuntu topics.\n> > \n> > Then I somehow found that another job is broken. And this patch\n> > seems to make it work. Not knowing how it got broken is\n> > unsatisfactory, though.\n> \n> I can hazard a guess (or several)...\n> \n> It is likely that the image pointed to by the fedora:latest\n> tag has moved from fedora 41 to 42, which was released\n> today. The fedora 41 container images have awk installed\n> while the fedora 42 images do not. That change is, I\n> suspect, just part of reducing the size of the base\n> container images.\n\nThank you for that excellent and thorough insight. This is indeed the case\nin the instances I looked at (for example, Git for Windows' `shears/main`\nbranch was using F42 most recently, and failing:\nhttps://github.com/git-for-windows/git/actions/runs/14485163916/job/40629342280#step:2:38\nwhereas the preceding build used F41 and succeeded:\nhttps://github.com/git-for-windows/git/actions/runs/14473893559/job/40594656287#step:2:37).\n\n> In both AlmaLinux and Fedora (as well as other RHEL\n> derivatives/relatives), awk is provided by the gawk package.\n> \n> On Fedora, `dnf install awk` uses the package filelist data\n> to determine that /usr/bin/awk is provided by gawk and\n> installs gawk as a result.\n> \n> On AlmaLinux (8 & 9, by my quick testing), that is not the\n> case and you'd need to use `dnf install gawk` or `dnf\n> install '*bin/awk'` to get it installed. Though awk is\n> included in the current AlmaLinux 8 and 9 images, so it\n> isn't strictly needed. But it's probably better to be\n> explicit that we need it installed, as a defense against\n> some future change to the AlmaLinux container removing awk.\n> \n> Using gawk (or even '*bin/awk') would likely be clearer.\n> That avoids relying on 1) a case statement to set an awk\n> variable; and 2) the different behavior of older and newer\n> releases of dnf with respect to the install argument\n> matching on the names of binaries provided by the package.\n> \n> But that's certainly subjective. ;)\n\nThis reasoning is very convincing, so I allowed myself to turn that into\nthe following commit (which fixed the build:\nhttps://github.com/git-for-windows/git/actions/runs/14485317015/job/40629741228):\n\n-- snipsnap --\nFrom 4890df8bf1c1311ef0c3b5a92412ccc4f833f944 Mon Sep 17 00:00:00 2001\nFrom: Johannes Schindelin \nDate: Wed, 16 Apr 2025 07:17:24 +0200\nSubject: [PATCH] ci(pedantic): ensure that awk is installed\n\nThe image pointed to by the fedora:latest tag has moved from fedora 41\nto 42. The fedora 41 container images have awk installed while the\nfedora 42 images do not. That change is most likely just part of\nreducing the size of the base container images.\n\nIn both AlmaLinux and Fedora (as well as other RHEL\nderivatives/relatives), awk is provided by the gawk package.\n\nOn Fedora, `dnf install awk` would work, but for unintended reasons! It\nuses the package filelist data to determine that /usr/bin/awk is\nprovided by gawk and installs gawk as a result.\n\nOn AlmaLinux (8 & 9, by my quick testing), that is not the case and\nyou'd need to use `dnf install gawk` or `dnf install '*bin/awk'` to get\nit installed. Having said that, awk _is_ included in the current\nAlmaLinux 8 and 9 images, so it isn't strictly needed. But it's\nprobably better to be explicit that we need it installed, as a defense\nagainst some future change to the AlmaLinux container removing awk.\n\nUsing the package name \"gawk\" is the right thing to do.\n\nNote that even '*bin/awk' would have worked, but it is less specific.\nAnd who knows, maybe in the far future a BSD variant of awk is offered,\ntoo, and would then cause ambiguities. Best to avoid that.\n\nSuggested-by: Todd Zullinger \nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 8700c0f2924d..be9ba5e30a47 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -31,7 +31,7 @@ alpine-*)\n \t;;\n fedora-*|almalinux-*)\n \tdnf -yq update >/dev/null &&\n-\tdnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\tdnf -yq install shadow-utils sudo make gcc findutils diffutils perl python3 gawk gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n \t;;\n ubuntu-*|i386/ubuntu-*|debian-*)\n \t# Required so that apt doesn't wait for user input on certain packages.\n-- \n\n"},{"id":"516287","messageId":"xmqqzfggi4pu.fsf@gitster.g","threadId":"63301","inReplyTo":"81209cfd-049e-c853-9c4a-2cf9803e56a4@gmx.de","subject":"Re: [RFH] fedora: pedantic fails complaining that awk is missing","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-04-16T14:03:09Z","receivedAt":"2025-04-16T14:03:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> This reasoning is very convincing, so I allowed myself to turn that into\n> the following commit (which fixed the build:\n> https://github.com/git-for-windows/git/actions/runs/14485317015/job/40629741228):\n>\n> -- snipsnap --\n> From 4890df8bf1c1311ef0c3b5a92412ccc4f833f944 Mon Sep 17 00:00:00 2001\n> From: Johannes Schindelin \n> Date: Wed, 16 Apr 2025 07:17:24 +0200\n> Subject: [PATCH] ci(pedantic): ensure that awk is installed\n\nSuperb. Thanks for working well together.\n\n>\n> The image pointed to by the fedora:latest tag has moved from fedora 41\n> to 42. The fedora 41 container images have awk installed while the\n> fedora 42 images do not. That change is most likely just part of\n> reducing the size of the base container images.\n>\n> In both AlmaLinux and Fedora (as well as other RHEL\n> derivatives/relatives), awk is provided by the gawk package.\n>\n> On Fedora, `dnf install awk` would work, but for unintended reasons! It\n> uses the package filelist data to determine that /usr/bin/awk is\n> provided by gawk and installs gawk as a result.\n>\n> On AlmaLinux (8 & 9, by my quick testing), that is not the case and\n> you'd need to use `dnf install gawk` or `dnf install '*bin/awk'` to get\n> it installed. Having said that, awk _is_ included in the current\n> AlmaLinux 8 and 9 images, so it isn't strictly needed. But it's\n> probably better to be explicit that we need it installed, as a defense\n> against some future change to the AlmaLinux container removing awk.\n>\n> Using the package name \"gawk\" is the right thing to do.\n\n> Note that even '*bin/awk' would have worked, but it is less specific.\n> And who knows, maybe in the far future a BSD variant of awk is offered,\n> too, and would then cause ambiguities. Best to avoid that.\n\nThe last two paragraphs may want to be rephrased.\n\nThe using scripts are saying \"awk\", not \"gawk\", so there is no\nspecific reason to avoid mawk or other implementations at least for\nnow. As long as installing one of them makes sure that end-user\nscripts that invoke \"awk\" ends up invoking the one that was\ninstalled, I have no objection to pick just one of them, but I do\nnot think \"we must be explicit to call for 'gawk' and not any other\nimplementation\" is what we want to say.\n\nInstead, we should say something like \"Because we know that on both\nof these distros, our scripts that call for 'awk' had been using\n'gawk' that was installed as part of the base image, let's make sure\nthat we explicitly install 'gawk'. If the image already has it, it\nwould be a no-op that does not cause breakage.\"\n\nSpecifically, we should not say we want to avoid BSD awk, unless we\nknow our awk scripts won't work with it and/or fixing a bug that was\ncaused by the incompatibilities.\n\n> Suggested-by: Todd Zullinger \n> Signed-off-by: Johannes Schindelin \n> ---\n\nThanks for not crediting my initial report ;-)\nThe patch text is as expected.\n"}]}