Re: [PATCH] ci: avoid building from the same commit in parallel

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Aug 23, 2023, 08:42 UTC
Message-ID: <1da763f3-60bf-a572-2c71-336b1fa5553d@gmx.de>
In-Reply-To: <xmqqjztnm6v0.fsf@gitster.g>

Hi Junio,

On Tue, 22 Aug 2023, Junio C Hamano wrote:

Show 15 quoted lines

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
> > Right, we'd need that `concurrency: ${{ github.sha }}` attribute on
> > the `config` job.
>
> That was my first thought, but I am not sure how it would work.
>
> Doesn't skip-if-redundant grab the workflow runs that have succeeded
> and then see if one for the same commit already exists?  If you used
> concurrency on the 'config', what gets serialized between two jobs
> for the same commit is only the 'config' phase, so 'master' may wait
> starting (because 'config' is what everybody else 'needs' it) while
> 'config' phase of 'main' runs, and then when it gets to the turn of
> 'config' phase of 'master', it would not find the run for the same
> commit being done for 'main' completed yet, would it?

Yes, that's true.

But there is a silver lining: the `concurrency` can not only be specified on the job level, but also on the workflow run level.

I tested this, and present the corresponding patch at the end of this mail.

Show 8 quoted lines

> > BTW there is another caveat. According to the documentation, if a job
> > is queued while another job is already queued, that other job is
> > canceled in favor of the latest one.
>
> Yes, that was the impression I got; your second one will wait (so
> you need a working skip-if-redundant to turn it into noop), but the
> third and subsequent ones are discarded without starting, which
> unfortunately is what we may want to see happen.

It's actually the last one that is still pending, while the intermediate ones will be canceled before they are started (see also the attached screenshot). The message that is shown in the web UI reads like this:

 𐤈 CI: .github#L1
   Canceling since a higher priority waiting request for '<SHA>' exists

See https://github.com/dscho/git/actions/runs/5948890677 for an example.

Here is the patch:

-- snipsnap --
From: Junio C Hamano <gitster@pobox.com>
Date: Mon, 21 Aug 2023 17:31:26 -0700
Subject: [PATCH] ci: avoid building from the same commit in parallel

At times, we may need to push the same commit to multiple branches in the same push. Rewinding 'next' to rebuild on top of 'master' soon after a release is such an occasion. Making sure 'main' stays in sync with 'master' to help those who expect that primary branch of the project is named either of these is another.

We already use the branch name as a "concurrency group" key, but that does not address the situation illustrated above.

Let's introduce another `concurrency` attribute, using the commit hash as the concurrency group key, on the workflow run level, to address this. This will hold any workflow run in the queued state when there is already a workflow run targeting the same commit, until that latter run completed. The `skip-if-redundant` check of the second run will then have a chance to see whether the first run succeeded.

The only caveat with this strategy is that only one workflow run will be kept in the queued state by the `concurrency` feature: if another run targeting the same commit is triggered, the previously-queued run will be canceled. Considering the benefit, this seems the smaller price to pay than to overload Git's build agent pool with undesired workflow runs.

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 .github/workflows/main.yml | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 30492eacddf..1739a0278dc 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -5,6 +5,19 @@ on: [push, pull_request]
 env:
   DEVELOPER: 1

+# If more than one workflow run is triggered for the very same commit hash
+# (which happens when multiple branches pointing to the same commit), only
+# the first one is allowed to run, the second will be kept in the "queued"
+# state. This allows a successful completion of the first run to be reused
+# in the second run via the `skip-if-redundant` logic in the `config` job.
+#
+# The only caveat is that if a workflow run is triggered for the same commit
+# hash that another run is already being held, that latter run will be
+# canceled. For more details about the `concurrency` attribute, see:
+# https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#concurrency
+concurrency:
+  group: ${{ github.sha }}
+
 jobs:
   ci-config:
     name: config
--
2.42.0.rc2.windows.1