Re: [PATCH 2/2] ci: match Linux jobs to available CPUs

From: Tamir Duberstein <tamird@gmail.com>
Date: Sep 25, 2026, 15:03 UTC
Message-ID: <CAJ-ks9nQF2E9Pp-mXq8VAo=m0ZUi4Am46ysSH9zU8cPK4V-Geg@mail.gmail.com>
In-Reply-To: <arS_nA3g-on2RdIL@pks.im>

On Thu, Sep 24, 2026 at 2:13 AM Patrick Steinhardt <ps@pks.im> wrote:

Show 8 quoted lines

>
> On Wed, Sep 23, 2026 at 01:13:29PM -0400, Tamir Duberstein wrote:
> > GitHub Actions runs ten Make test suites concurrently even on private
> > Linux runners with two CPUs. Pull request runs enable the long tests.
> > These runs hit ENOSPC while multiple multi-gigabyte clone and repack
> > fixtures were active.
>
> Again, a link would be appreciated that demonstrates this.

Yeah, sorry about that. Again, I suspect GitHub runners on private repos (2 CPUs) make this worse than public repos (4 CPUs).

Show 7 quoted lines

> > Use nproc to choose Make and prove parallelism, as the GitLab CI path
> > already does. This reduces overlapping fixtures on small Linux runners
> > while keeping the long tests enabled.
>
> It may avoid overlapping fixtures. But what does CI runtime look like
> before and after this change? Does it improve? Does it regress? Would it
> maybe make sense to oversubscribe at least a bit?

I ran a bunch of tests of this in GitHub and the disappointing answer is that it's not clear what the right choice is:

                    Fixed 10   CPU count   2x CPU count
Linux Make             278.9       273.9          259.9
macOS Make              94.5       119.1           99.8
Windows Make           102.2       103.8          100.8

Show 17 quoted lines

>
> > diff --git a/ci/lib.sh b/ci/lib.sh
> > index c6ccbf8c17..0855026dad 100755
> > --- a/ci/lib.sh
> > +++ b/ci/lib.sh
> > @@ -228,6 +228,10 @@ then
> >
> >       GIT_TEST_OPTS="--github-workflow-markup"
> >       JOBS=10
> > +     if test linux = "$CI_OS_NAME"
> > +     then
> > +             JOBS=$(nproc)
> > +     fi
>
> Makes me wonder whether we should have the same logic on both GitLab and
> GitHub going forward. There probably isn't a good reason why these two
> should differ from one another.

Agreed. I have rewritten this patch to use the same logic across CI providers (1x CPU count) in v2. I'll leave tuning (e.g. moving to 2x) to a future change.

>
> Patrick