threads / patch / 64492

patchci(dockerized): do show the result of failing tests again

Subject: [PATCH] ci(dockerized): do show the result of failing tests again

## tl;dr

9 messages between Nov 17, 2025 and Nov 29, 2025. Diffs are folded; open one to read it.

replies: 8people: 4as markdown or json

Johannes Schindelin via GitGitGadget· Nov 17, 2025, 17:04 UTC · lore
From: Johannes Schindelin <johannes.schindelin@gmx.de>

The quality of tests/test suites does not show as much when there are no breakages as in the amount of time required after bugs trigger test failures before the bugs can be identified, analyzed and resolved.

As such, it is an unfortunate side effect of 2a21098b98a (github: adapt containerized jobs to be rootless, 2025-01-10) that the output of failed test cases, which was shown before that change directly in the build logs, is now no longer shown at all.

The reason is a side effect of trying to run the build and the tests with permissions other than the `root` user, but without providing the prerequisite permissions to signal what tests failed and whose output hence needs to be included in the logs.

The way this signaling works is for the workflow to write into special-purpose files whose path is specific to the current workflow step and which can be accessed via the `$GITHUB_ENV` environment variable, which differs between workflow steps. It is this file that is missing write permission for the `builder` user that was introduced in above-mentioned commit.

The solution is simple: make the file world-writable.

Technically, this write permission should be removed after the step has completed, if proper security practices were to be upheld, but since nothing uses that file again, it does not matter, and the fix is more succinct this way.

This commit is best viewed with `--color-words`.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
    ci(dockerized): do show the result of failing tests again
    
    It has become quite hard to debug CI failures when they happen in one of
    the Dockerized jobs, as the actual test failures are now hidden. This
    was most likely an oversight when 2a21098b98a (github: adapt
    containerized jobs to be rootless, 2025-01-10) was merged in 2bf3c7fab19
    (Merge branch 'ps/ci-misc-updates', 2025-02-06), v2.49.0-rc0~55, and I
    had reported this as a regression in
    https://lore.kernel.org/git/e45b9487-b3ae-ed85-fd07-c92cfbf47cbb@gmx.de/.
    Seeing no movement on my report, and having the pressure of
    newly-failing tests during the v2.52.0-rc0 rebase of Git for Windows, I
    was kind of forced into fixing this in Git for Windows. Here I upstream
    the fix.
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2003%2Fdscho%2Ffix-failure-reporting-in-dockerized-ci-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2003/dscho/fix-failure-reporting-in-dockerized-ci-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2003
 .github/workflows/main.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Show changes to .github/workflows/main.yml +1 −1
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 816d5a34c4..ca7cc2984f 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -433,7 +433,7 @@ jobs:
     - run: ci/install-dependencies.sh
     - run: useradd builder --create-home
     - run: chown -R builder .
-    - run: sudo --preserve-env --set-home --user=builder ci/run-build-and-tests.sh
+    - run: chmod o+w $GITHUB_ENV && sudo --preserve-env --set-home --user=builder ci/run-build-and-tests.sh
     - name: print test failures
       if: failure() && env.FAILED_TEST_ARTIFACTS != ''
       run: sudo --preserve-env --set-home --user=builder ci/print-test-failures.sh

base-commit: 621415c8b5371a4734315232a780dd8282f6fe4f
-- 
gitgitgadget
Junio C Hamano· Nov 17, 2025, 18:28 UTC · re: Johannes Schindelin via GitGitGadget · lore

Re: [PATCH] ci(dockerized): do show the result of failing tests again

"Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com> writes:

Show 24 quoted lines
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> The quality of tests/test suites does not show as much when there are no
> breakages as in the amount of time required after bugs trigger test
> failures before the bugs can be identified, analyzed and resolved.
>
> As such, it is an unfortunate side effect of 2a21098b98a (github: adapt
> containerized jobs to be rootless, 2025-01-10) that the output of failed
> test cases, which was shown before that change directly in the build
> logs, is now no longer shown at all.
>
> The reason is a side effect of trying to run the build and the tests
> with permissions other than the `root` user, but without providing the
> prerequisite permissions to signal what tests failed and whose output
> hence needs to be included in the logs.
>
> The way this signaling works is for the workflow to write into
> special-purpose files whose path is specific to the current workflow
> step and which can be accessed via the `$GITHUB_ENV` environment
> variable, which differs between workflow steps. It is this file that is
> missing write permission for the `builder` user that was introduced in
> above-mentioned commit.
>
> The solution is simple: make the file world-writable.

I expected to see a+w not o+w from this statement; as long as it works I have no strong objections, but if I saw o+w without the above explanation I would probably have wondered who are in the group that we do not want this file touched by.

Show 9 quoted lines
> Technically, this write permission should be removed after the step has
> completed, if proper security practices were to be upheld, but since
> nothing uses that file again, it does not matter, and the fix is more
> succinct this way.
>
> This commit is best viewed with `--color-words`.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
Show 20 quoted lines
> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2003%2Fdscho%2Ffix-failure-reporting-in-dockerized-ci-v1
> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2003/dscho/fix-failure-reporting-in-dockerized-ci-v1
> Pull-Request: https://github.com/gitgitgadget/git/pull/2003
>
>  .github/workflows/main.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
> index 816d5a34c4..ca7cc2984f 100644
> --- a/.github/workflows/main.yml
> +++ b/.github/workflows/main.yml
> @@ -433,7 +433,7 @@ jobs:
>      - run: ci/install-dependencies.sh
>      - run: useradd builder --create-home
>      - run: chown -R builder .
> -    - run: sudo --preserve-env --set-home --user=builder ci/run-build-and-tests.sh
> +    - run: chmod o+w $GITHUB_ENV && sudo --preserve-env --set-home --user=builder ci/run-build-and-tests.sh
>      - name: print test failures
>        if: failure() && env.FAILED_TEST_ARTIFACTS != ''
>        run: sudo --preserve-env --set-home --user=builder ci/print-test-failures.sh
Thanks.  Will apply.
Junio C Hamano· Nov 23, 2025, 02:41 UTC · re: Junio C Hamano · lore

Re: [PATCH] ci(dockerized): do show the result of failing tests again

Junio C Hamano <gitster@pobox.com> writes:
Show 12 quoted lines
>> The solution is simple: make the file world-writable.
>
> I expected to see a+w not o+w from this statement; as long as it
> works I have no strong objections, but if I saw o+w without the
> above explanation I would probably have wondered who are in the
> group that we do not want this file touched by.
> ...
>>      - run: useradd builder --create-home
>>      - run: chown -R builder .
>> -    - run: sudo --preserve-env --set-home --user=builder ci/run-build-and-tests.sh
>> +    - run: chmod o+w $GITHUB_ENV && sudo --preserve-env --set-home --user=builder ci/run-build-and-tests.sh
>>      - name: print test failures

Unless I hear that "user X belongs to the same group as our user that runs 'chmod' on $GITHUB_ENV, and we do not want that user to be writing into the file", I'll amend the patch text to match the "solution" described in the proposed log message to "chmod a+w", before we mark the topic for 'next'.

Thanks.
Jeff King· Nov 18, 2025, 09:36 UTC · re: Johannes Schindelin via GitGitGadget · lore

Re: [PATCH] ci(dockerized): do show the result of failing tests again

On Mon, Nov 17, 2025 at 05:04:24PM +0000, Johannes Schindelin via GitGitGadget wrote:
Show 6 quoted lines
> The way this signaling works is for the workflow to write into
> special-purpose files whose path is specific to the current workflow
> step and which can be accessed via the `$GITHUB_ENV` environment
> variable, which differs between workflow steps. It is this file that is
> missing write permission for the `builder` user that was introduced in
> above-mentioned commit.

Thanks for fixing this. It bit me recently, but I hadn't had time to look at it yet.

BTW, I ran into a similar issue (no useful output from a failed test) in the windows-meson job, but the cause is totally different there:

  https://lore.kernel.org/git/20251118093221.GA530337@coredump.intra.peff.net/
-Peff
Elijah Newren· Nov 25, 2025, 06:15 UTC · re: Johannes Schindelin via GitGitGadget · lore

Re: [PATCH] ci(dockerized): do show the result of failing tests again

On Mon, Nov 17, 2025 at 9:17 AM Johannes Schindelin via GitGitGadget <gitgitgadget@gmail.com> wrote:

Show 6 quoted lines
>
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> The quality of tests/test suites does not show as much when there are no
> breakages as in the amount of time required after bugs trigger test
> failures before the bugs can be identified, analyzed and resolved.

I found this paragraph hard to parse. After re-reading a couple times, does the following convey the same meaning?:

The quality of tests and test suites is most apparent not when everything passes, but in how quickly bugs can be identified, analyzed, and resolved after test failures occur.

Junio C Hamano· Nov 25, 2025, 14:32 UTC · re: Elijah Newren · lore

Re: [PATCH] ci(dockerized): do show the result of failing tests again

Elijah Newren <newren@gmail.com> writes:
Show 15 quoted lines
> On Mon, Nov 17, 2025 at 9:17 AM Johannes Schindelin via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
>>
>> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>>
>> The quality of tests/test suites does not show as much when there are no
>> breakages as in the amount of time required after bugs trigger test
>> failures before the bugs can be identified, analyzed and resolved.
>
> I found this paragraph hard to parse.  After re-reading a couple
> times, does the following convey the same meaning?:
>
> The quality of tests and test suites is most apparent not when
> everything passes, but in how quickly bugs can be identified,
> analyzed, and resolved after test failures occur.

FWIW, I had the same "I cannot quite figure out how this paragraph really wants to help readers by saying this" reaction to the paragraph. Your rewrite finally helped me understand the intention (if that is what the originall wanted to say, that is).

Thanks.
Johannes Schindelin· Nov 25, 2025, 17:40 UTC · re: Elijah Newren · lore

Re: [PATCH] ci(dockerized): do show the result of failing tests again

Hi Elijah,
On Mon, 24 Nov 2025, Elijah Newren wrote:
Show 15 quoted lines
> On Mon, Nov 17, 2025 at 9:17 AM Johannes Schindelin via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
> >
> > From: Johannes Schindelin <johannes.schindelin@gmx.de>
> >
> > The quality of tests/test suites does not show as much when there are no
> > breakages as in the amount of time required after bugs trigger test
> > failures before the bugs can be identified, analyzed and resolved.
> 
> I found this paragraph hard to parse.  After re-reading a couple
> times, does the following convey the same meaning?:
> 
> The quality of tests and test suites is most apparent not when
> everything passes, but in how quickly bugs can be identified,
> analyzed, and resolved after test failures occur.
Yes, this reflects what I tried to say.

Ciao, Johannes

Junio C Hamano· Nov 25, 2025, 19:27 UTC · re: Johannes Schindelin · lore

Re: [PATCH] ci(dockerized): do show the result of failing tests again

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
Show 21 quoted lines
> Hi Elijah,
>
> On Mon, 24 Nov 2025, Elijah Newren wrote:
>
>> On Mon, Nov 17, 2025 at 9:17 AM Johannes Schindelin via GitGitGadget
>> <gitgitgadget@gmail.com> wrote:
>> >
>> > From: Johannes Schindelin <johannes.schindelin@gmx.de>
>> >
>> > The quality of tests/test suites does not show as much when there are no
>> > breakages as in the amount of time required after bugs trigger test
>> > failures before the bugs can be identified, analyzed and resolved.
>> 
>> I found this paragraph hard to parse.  After re-reading a couple
>> times, does the following convey the same meaning?:
>> 
>> The quality of tests and test suites is most apparent not when
>> everything passes, but in how quickly bugs can be identified,
>> analyzed, and resolved after test failures occur.
>
> Yes, this reflects what I tried to say.

So, do you mind if I locally amended the log message, or should we expect an updated patch sent to the list? For a small thing like this, either is fine by me.

Thanks.
Johannes Schindelin· Nov 29, 2025, 18:31 UTC · re: Junio C Hamano · lore

Re: [PATCH] ci(dockerized): do show the result of failing tests again

Hi Junio,
On Tue, 25 Nov 2025, Junio C Hamano wrote:
Show 25 quoted lines
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > On Mon, 24 Nov 2025, Elijah Newren wrote:
> >
> >> On Mon, Nov 17, 2025 at 9:17 AM Johannes Schindelin via GitGitGadget
> >> <gitgitgadget@gmail.com> wrote:
> >> >
> >> > From: Johannes Schindelin <johannes.schindelin@gmx.de>
> >> >
> >> > The quality of tests/test suites does not show as much when there are no
> >> > breakages as in the amount of time required after bugs trigger test
> >> > failures before the bugs can be identified, analyzed and resolved.
> >> 
> >> I found this paragraph hard to parse.  After re-reading a couple
> >> times, does the following convey the same meaning?:
> >> 
> >> The quality of tests and test suites is most apparent not when
> >> everything passes, but in how quickly bugs can be identified,
> >> analyzed, and resolved after test failures occur.
> >
> > Yes, this reflects what I tried to say.
> 
> So, do you mind if I locally amended the log message, or should we
> expect an updated patch sent to the list?  For a small thing like
> this, either is fine by me.

Sure, I saw that you amended the log message and also changed the `chmod` to be a bit more robust.

Ciao, Johannes

← back to recent threads