win+Meson test failures due to ps/build-meson-fixes-0130 topic?

> As of yesterday, 'next' started failing all the "win+Meson test"
> jobs.
> 
> https://github.com/git/git/actions/runs/13506034920
> 
> I had to do one of my least favourite activities working on Git:
> bisecting CI breakage on a platform I have no idea how well it
> works.
> 
> Here is the full "stack" of changes, it contains all the topics
> merged between 'master' and 'next', plus a single change that
> butchers .github/workflows/main.yml to leave only three jobs
> (ci-config, windows-meson-build, and windows-meson-test).
> 
>     3a7e83b527 Merge branch 'ps/build-meson-fixes-0130'
>     e59193e984 Merge branch 'ek/mingw-rename-symlink'
>     a209cf1427 Merge branch 'po/meson-perl-fix'
>     9f1e73677c Merge branch 'ua/os-version-capability'
>     82f01cb375 Merge branch 'ms/rename-match-name-with-pattern'
>     8b04381049 Merge branch 'ad/set-default-target-in-makefiles'
>     2cab1bd7d0 Merge branch 'pw/merge-tree-stdin-deadlock-fix'
>     92b43de9df Merge branch 'mh/doc-commit-title-not-subject'
>     685c150623 Merge branch 'bc/diff-reject-empty-arg-to-pickaxe'
>     c5093fc2d8 Merge branch 'tb/new-make-fix'
>     9091607e95 Merge branch 'da/xdiff-w-sign-compare-workaround'
>     01e86410fe GitHub CI: trim and slim for win+Meson test breakage hunting
>     2d2a71ce85 The thirteenth batch
> 
> At 01e86410fe and e59193e984, CI seems to pass.
> 
> https://github.com/git/git/actions/runs/13528755524
> 
> With ps/build-meson-fixes-0130 at 3a7e83b527, all the "win+Meson
> test" jobs fail.  I suspect that the series does not break
> individual tests but the breakage is in somewhere more
> framework-ish?  I didn't look into individual test job failures.
> 
> https://github.com/git/git/actions/runs/13529323827
> 
> I'll also note that in one of my earlier attempts (which had
> ek/mingw-rename-symlink merged before po/meson-perl-fix), one of the
> 10 "win+Meson test" jobs failed ("win+Meson test (7)" to be exact).
> Rerunning the failed test succeeds, so there must be something flaky
> in t0610-reftable-basics test that is exposed by this change,
> perhaps?
> 
>   https://github.com/git/git/actions/runs/13528071468/attempts/1
>   https://github.com/git/git/actions/runs/13528071468/attempts/2
> 
> As the ps/build-meson-fixes-0130 topic has 13 changes, and each
> iteration in the above exercise took somewhere around 20 minutes, I
> won't dive into the topic myself to identify which one of these
> individual topics causes the breakage.  This however gave me enough
> to go on to today's 'master' integration (mainly, by mark the topic
> to be held in 'next' until this gets resolved, and deal with the
> other topics).

> > I'll also note that in one of my earlier attempts (which had
> > ek/mingw-rename-symlink merged before po/meson-perl-fix), one of the
> > 10 "win+Meson test" jobs failed ("win+Meson test (7)" to be exact).
> > Rerunning the failed test succeeds, so there must be something flaky
> > in t0610-reftable-basics test that is exposed by this change,
> > perhaps?
> > 
> >   https://github.com/git/git/actions/runs/13528071468/attempts/1
> >   https://github.com/git/git/actions/runs/13528071468/attempts/2
> > 
> > As the ps/build-meson-fixes-0130 topic has 13 changes, and each
> > iteration in the above exercise took somewhere around 20 minutes, I
> > won't dive into the topic myself to identify which one of these
> > individual topics causes the breakage.  This however gave me enough
> > to go on to today's 'master' integration (mainly, by mark the topic
> > to be held in 'next' until this gets resolved, and deal with the
> > other topics).
> 
> Hm, interesting. I didn't notice those issues in our CI. Thanks for
> diving into it, I'll figure out what has happened here.

> On Tue, Feb 25, 2025 at 11:45:47AM -0800, Junio C Hamano wrote:
> > As of yesterday, 'next' started failing all the "win+Meson test"
> > jobs.
> > 
> > https://github.com/git/git/actions/runs/13506034920
> > 
> > I had to do one of my least favourite activities working on Git:
> > bisecting CI breakage on a platform I have no idea how well it
> > works.
> > 
> > Here is the full "stack" of changes, it contains all the topics
> > merged between 'master' and 'next', plus a single change that
> > butchers .github/workflows/main.yml to leave only three jobs
> > (ci-config, windows-meson-build, and windows-meson-test).
> > 
> >     3a7e83b527 Merge branch 'ps/build-meson-fixes-0130'
> >     e59193e984 Merge branch 'ek/mingw-rename-symlink'
> >     a209cf1427 Merge branch 'po/meson-perl-fix'
> >     9f1e73677c Merge branch 'ua/os-version-capability'
> >     82f01cb375 Merge branch 'ms/rename-match-name-with-pattern'
> >     8b04381049 Merge branch 'ad/set-default-target-in-makefiles'
> >     2cab1bd7d0 Merge branch 'pw/merge-tree-stdin-deadlock-fix'
> >     92b43de9df Merge branch 'mh/doc-commit-title-not-subject'
> >     685c150623 Merge branch 'bc/diff-reject-empty-arg-to-pickaxe'
> >     c5093fc2d8 Merge branch 'tb/new-make-fix'
> >     9091607e95 Merge branch 'da/xdiff-w-sign-compare-workaround'
> >     01e86410fe GitHub CI: trim and slim for win+Meson test breakage hunting
> >     2d2a71ce85 The thirteenth batch
> > 
> > At 01e86410fe and e59193e984, CI seems to pass.
> > 
> > https://github.com/git/git/actions/runs/13528755524
> > 
> > With ps/build-meson-fixes-0130 at 3a7e83b527, all the "win+Meson
> > test" jobs fail.  I suspect that the series does not break
> > individual tests but the breakage is in somewhere more
> > framework-ish?  I didn't look into individual test job failures.
> > 
> > https://github.com/git/git/actions/runs/13529323827
> > 
> > I'll also note that in one of my earlier attempts (which had
> > ek/mingw-rename-symlink merged before po/meson-perl-fix), one of the
> > 10 "win+Meson test" jobs failed ("win+Meson test (7)" to be exact).
> > Rerunning the failed test succeeds, so there must be something flaky
> > in t0610-reftable-basics test that is exposed by this change,
> > perhaps?
> > 
> >   https://github.com/git/git/actions/runs/13528071468/attempts/1
> >   https://github.com/git/git/actions/runs/13528071468/attempts/2
> > 
> > As the ps/build-meson-fixes-0130 topic has 13 changes, and each
> > iteration in the above exercise took somewhere around 20 minutes, I
> > won't dive into the topic myself to identify which one of these
> > individual topics causes the breakage.  This however gave me enough
> > to go on to today's 'master' integration (mainly, by mark the topic
> > to be held in 'next' until this gets resolved, and deal with the
> > other topics).
> 
> Hm, interesting. I didn't notice those issues in our CI. Thanks for
> diving into it, I'll figure out what has happened here.

> > Hm, interesting. I didn't notice those issues in our CI. Thanks for
> > diving into it, I'll figure out what has happened here.
> 
> Okay, the issue was in 163a3e97ac1 (meson: simplify use of the
> common-main library, 2025-01-30). In that commit we stopped using the
> common-main dependency for a couple of binaries, including
> git-remote-https and others. What I didn't realize though is that this
> also made us not use the linker arguments anymore, but those are
> required on Windows when linking an executable.

> On Wed, Feb 26, 2025 at 02:46:53AM -0500, Jeff King wrote:
> 
> > I'd imagine that if I did a complete "rm -rf build && meson setup build"
> > it would work. But it's pretty lousy to have to do that preemptively
> > during a bisection.
> 
> Looks like this runs even deeper. On my Linux machine:
> 
>   $ git checkout origin/next
>   $ meson setup build
>   [...ok...]
>   $ meson compile -C build
>   [...ok...]
> 
>   $ git checkout origin/master
>   $ meson compile -C build
>   [...]
>   ../meson.build:215:35: ERROR: The `!=` operator of array does not accept objects of type str ()
>   FAILED: build.ninja
>   /usr/bin/meson --internal regenerate /home/peff/compile/git .
>   ninja: error: rebuilding 'build.ninja': subcommand failed
> 
>   $ rm -rf build
>   $ meson setup build
>   $ meson compile -C build
>   [...ok again...]
> 
> I'm hoping you can tell me I'm holding it wrong, and there's some way to
> do an incremental build when crossing these sorts of boundaries (without
> blowing away all of the build products).
> 
> I do at least use ccache which makes the "rm -rf" case a little less
> painful.

> > I'm hoping you can tell me I'm holding it wrong, and there's some way to
> > do an incremental build when crossing these sorts of boundaries (without
> > blowing away all of the build products).
> > 
> > I do at least use ccache which makes the "rm -rf" case a little less
> > painful.
> 
> You're not holding it wrong. The problem is that this patch series
> introduces a backwards-incompatible change to the "sane_tools_path"
> option as it changes it from a string to an array, which makes Meson
> hiccup. Once Meson is not considered to be experimental anymore we
> should refrain from changing option types going forward, but until then
> I consider it to be okayish to break things like this.
> 
> From my point of view we can treat Meson as stable as soon as Git v2.49
> is out. So from thereon, I will make sure to not include backwards
> incompatible changes anymore.

> Okay, the issue was in 163a3e97ac1 (meson: simplify use of the
> common-main library, 2025-01-30). In that commit we stopped using the
> common-main dependency for a couple of binaries, including
> git-remote-https and others. What I didn't realize though is that this
> also made us not use the linker arguments anymore, but those are
> required on Windows when linking an executable.
>
> I've kicked off CI jobs on GitHub to verify that the revised version of
> this patch series works alright. Once those tests pass I'll send another
> version of this patch series.

> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Okay, the issue was in 163a3e97ac1 (meson: simplify use of the
> > common-main library, 2025-01-30). In that commit we stopped using the
> > common-main dependency for a couple of binaries, including
> > git-remote-https and others. What I didn't realize though is that this
> > also made us not use the linker arguments anymore, but those are
> > required on Windows when linking an executable.
> >
> > I've kicked off CI jobs on GitHub to verify that the revised version of
> > this patch series works alright. Once those tests pass I'll send another
> > version of this patch series.
> 
> Thanks.  What is a bit more worrying is that you earlier said your
> other CI didn't notice anything funny.  Is the reason for the
> non-breakage also known (like: the two CI jobs build things
> differently)?