v2, 11 partsGIT-VERSION-GEN: simplify computing the dirty marker

Hi,

this small patch series backfills in a couple of missing features into Meson. It also improves test coverage of our Meson-based CI jobs so that we compile with Meson with Visual Studio and compile fuzzers. CI runs for GitLab and GitHub can be found at [1] and [2], respectively.

The series is built on top of fbe8d3079d (Git 2.48, 2025-01-10) with ps/meson-weak-sha1-build at 6a0ee54f9a (meson: provide a summary of configured backends, 2024-12-30) merged into it.

Changes in v2:
  - Consistently use `meson.has_header_symbol()` to fix warnings for
    features not yet available in Meson 0.61, which is our minimum
    required version.
  - Add another patch that makes use use `--fatal-meson-warnings` so
    that such warnings will cause the build to fail.
  - Fix a bug that made GIT-VERSION-GEN always return tags as version.
  - Adapt the approach we use to populate the project and distribution
    tarball versions.
  - Link to v1: https://lore.kernel.org/r/20250113-b4-pks-meson-additions-v1-0-97f6a93f691d@pks.im

Thanks!

Patrick

[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/280 [2]: https://github.com/git/git/pull/1870

---
Patrick Steinhardt (11):
      GIT-VERSION-GEN: simplify computing the dirty marker
      GIT-VERSION-GEN: allow running without input and output files
      meson: populate project version via GIT-VERSION-GEN
      meson: fix dependencies for generated headers
      meson: wire up development environments
      meson: wire up generation of distribution archive
      meson: wire up fuzzers
      meson: make the CSPRNG backend configurable
      meson: fix compilation with Visual Studio
      ci: raise error when Meson generates warnings
      ci: wire up Visual Studio build with Meson

 .github/workflows/main.yml | 52 +++++++++++++++++++++++++++
 .gitlab-ci.yml             | 38 ++++++++++++++++++++
 GIT-VERSION-GEN            | 50 ++++++++++++++++----------
 ci/run-build-and-tests.sh  |  4 ++-
 meson.build                | 87 +++++++++++++++++++++++++++++++++++++---------
 meson_options.txt          |  4 +++
 oss-fuzz/meson.build       | 20 +++++++++++
 7 files changed, 219 insertions(+), 36 deletions(-)

Range-diff versus v1:

 1:  9cceb5db4d !  1:  a546dd43be GIT-VERSION-GEN: simplify computing the dirty marker
    @@ GIT-VERSION-GEN: then
      			test -f "$SOURCE_DIR"/.git;
      		} &&
     -		VN=$(git -C "$SOURCE_DIR" describe --match "v[0-9]*" HEAD 2>/dev/null) &&
    -+		VN=$(git -C "$SOURCE_DIR" describe --match --dirty "v[0-9]*" 2>/dev/null) &&
    ++		VN=$(git -C "$SOURCE_DIR" describe --dirty --match="v[0-9]*" 2>/dev/null) &&
      		case "$VN" in
      		*$LF*) (exit 1) ;;
     -		v[0-9]*)
 2:  00b9586887 <  -:  ---------- GIT-VERSION-GEN: move default version into a separate file
 -:  ---------- >  2:  3079bcc87a GIT-VERSION-GEN: allow running without input and output files
 -:  ---------- >  3:  aab1945e47 meson: populate project version via GIT-VERSION-GEN
 3:  a7712e6874 =  4:  782b9677c3 meson: fix dependencies for generated headers
 4:  0b30d76c7a =  5:  a9efaebe7b meson: wire up development environments
 5:  71081feccf !  6:  92913a6b20 meson: wire up generation of distribution archive
    @@ Metadata
      ## Commit message ##
         meson: wire up generation of distribution archive
     
    -    Meson knows to generate distribution archives via `meson dist`. Despite
    -    generating the archive itself, this target also knows to compile and
    -    execute tests from that archive, which helps to ensure that the result
    -    is an adequate drop-in replacement for the versioned project.
    +    Meson knows to generate distribution archives via `meson dist`. In
    +    addition to generating the archive itself, this target also knows to
    +    compile and execute tests from that archive, which helps to ensure that
    +    the result is an adequate drop-in replacement for the versioned project.
     
         While this already works as-is, one omission is that we don't propagate
         the commit that this is built from into the resulting archive. This can
    @@ Commit message
         version into the "version" file, which GIT-VERSION-GEN knows to read if
         present.
     
    -    Use GIT-VERSION-GEN itself to populate that file. There are two smallish
    -    gotchas:
    -
    -      - The script is executed in the build directory, not in the directory
    -        where we generate the archive. The target directory is propagated
    -        via the "MESON_DIST_ROOT" environment variable, but because we don't
    -        use a shell to execute GIT-VERSION-GEN we cannot populate the envvar
    -        into its arguments. We thus adapt GIT-VERSION-GEN to handle this for
    -        us.
    -
    -      - We use the "GIT-VERSION-FILE.in" template to generate the version,
    -        which contains a "GIT_VERSION=" prefix that we need to strip after
    -        reading the "version" file. We could avoid this extra logic if we
    -        used a template that only had the `@GIT_VERSION@` placeholder, but
    -        it would require us to add one more template file.
    +    Use GIT-VERSION-GEN to populate that file. As the script is executed in
    +    the build directory, not in the directory where we generate the archive,
    +    we have adapt it to honor the "MESON_DIST_ROOT" environment variable.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## GIT-VERSION-GEN ##
    -@@ GIT-VERSION-GEN: then
    - 	exit 1
    - fi
    +@@ GIT-VERSION-GEN: case "$2" in
    + esac
      
    + OUTPUT="$3"
     +if test -n "$OUTPUT" && test -n "$MESON_DIST_ROOT"
     +then
     +    OUTPUT="$MESON_DIST_ROOT/$OUTPUT"
     +fi
    -+
    - DEF_VER=$(cat "$SOURCE_DIR"/GIT-VERSION)
      
      # Protect us from reading Git version information outside of the Git directory
    -@@ GIT-VERSION-GEN: then
    - 	# then try git-describe, then default.
    - 	if test -f "$SOURCE_DIR"/version
    - 	then
    --		VN=$(cat "$SOURCE_DIR"/version) || VN="$DEF_VER"
    -+		VN=$(cat "$SOURCE_DIR"/version) && VN=${VN#GIT_VERSION=} || VN="$DEF_VER"
    - 	elif {
    - 			test -d "$SOURCE_DIR/.git" ||
    - 			test -d "${GIT_DIR:-.git}" ||
    + # in case it is not a repository itself, but embedded in an unrelated
     
      ## meson.build ##
     @@ meson.build: devenv.set('GIT_BUILD_DIR', meson.current_build_dir())
    @@ meson.build: devenv.set('GIT_BUILD_DIR', meson.current_build_dir())
     +  shell,
     +  meson.current_source_dir() / 'GIT-VERSION-GEN',
     +  meson.current_source_dir(),
    -+  meson.current_source_dir() / 'GIT-VERSION-FILE.in',
    ++  '--format=@GIT_VERSION@',
     +  'version',
     +)
     +
 6:  5dd6be9be7 =  7:  4bde8aee4c meson: wire up fuzzers
 7:  88e2935e52 !  8:  2f33403322 meson: make the CSPRNG backend configurable
    @@ meson.build: else
      endif
      
     -if compiler.has_header_symbol('stdlib.h', 'arc4random_buf')
    ++# Backends are ordered to reflect our preference for more secure and faster
    ++# ones over the ones that are less so.
     +if csprng_backend in ['auto', 'arc4random'] and compiler.has_header_symbol('stdlib.h', 'arc4random_buf', required: csprng_backend == 'arc4random')
        libgit_c_args += '-DHAVE_ARC4RANDOM'
     -elif compiler.has_header_symbol('bsd/stdlib.h', 'arc4random_buf')
    @@ meson.build: else
        libgit_c_args += '-DHAVE_ARC4RANDOM_BSD'
     -elif compiler.has_function('getrandom', prefix: '#include <sys/random.h>')
     +  csprng_backend = 'arc4random_bsd'
    -+elif csprng_backend in ['auto', 'getrandom'] and compiler.has_function('getrandom', prefix: '#include <sys/random.h>', required: csprng_backend == 'getrandom')
    ++elif csprng_backend in ['auto', 'getrandom'] and compiler.has_header_symbol('sys/random.h', 'getrandom', required: csprng_backend == 'getrandom')
        libgit_c_args += '-DHAVE_GETRANDOM'
     -elif compiler.has_function('getentropy', prefix: '#include <unistd.h>')
     +  csprng_backend = 'getrandom'
    -+elif csprng_backend in ['auto', 'getentropy'] and compiler.has_function('getentropy', prefix: '#include <unistd.h>', required: csprng_backend == 'getentropy')
    ++elif csprng_backend in ['auto', 'getentropy'] and compiler.has_header_symbol('unistd.h', 'getentropy', required: csprng_backend == 'getentropy')
        libgit_c_args += '-DHAVE_GETENTROPY'
     -elif compiler.has_function('RtlGenRandom', prefix: '#include <windows.h>\n#include <ntsecapi.h>')
     +  csprng_backend = 'getentropy'
    -+elif csprng_backend in ['auto', 'rtlgenrandom'] and compiler.has_function('RtlGenRandom', prefix: '#include <windows.h>\n#include <ntsecapi.h>', required: csprng_backend == 'rtlgenrandom')
    ++elif csprng_backend in ['auto', 'rtlgenrandom'] and compiler.has_header_symbol('ntsecapi.h', 'RtlGenRandom', prefix: '#include <windows.h>', required: csprng_backend == 'rtlgenrandom')
        libgit_c_args += '-DHAVE_RTLGENRANDOM'
     -elif openssl.found()
     +  csprng_backend = 'rtlgenrandom'
 8:  f6ee13d6dc <  -:  ---------- meson: fix compilation with Visual Studio
 -:  ---------- >  9:  d56a24e15a meson: fix compilation with Visual Studio
 -:  ---------- > 10:  c624f1d554 ci: raise error when Meson generates warnings
 9:  946c31f452 = 11:  c7f7e2309a ci: wire up Visual Studio build with Meson

--- base-commit: 35a417ddf0eab983e4d5eb69e628aa198114bb05 change-id: 20250107-b4-pks-meson-additions-055c16b3052b

The GIT-VERSION-GEN script computes the version that Git is being built from. When building from a commit with an unclean worktree it knows to append "-dirty" to that version to indicate that there were custom changes applied and that it isn't the exact same as that commit.

The dirtiness check is done manually via git-diff-index(1), which is somewhat puzzling though: we already use git-describe(1) to compute the version, which also knows to compute dirtiness via the "--dirty" flag. But digging back in history explains why: the "-dirty" suffix was added in 31e0b2ca81 (GIT 1.5.4.3, 2008-02-23), and git-describe(1) didn't yet have support for "--dirty" back then.

Refactor the script to use git-describe(1). Despite being simpler, it also results in a small speedup:

    Benchmark 1: git describe --dirty --match "v[0-9]*"
      Time (mean ± σ):      12.5 ms ±   0.3 ms    [User: 6.3 ms, System: 8.8 ms]
      Range (min … max):    12.0 ms …  13.5 ms    200 runs

    Benchmark 2: git describe --match "v[0-9]*" HEAD && git update-index -q --refresh && git diff-index --name-only HEAD --
      Time (mean ± σ):      17.9 ms ±   1.1 ms    [User: 8.8 ms, System: 14.4 ms]
      Range (min … max):    17.0 ms …  30.6 ms    148 runs

    Summary
      git describe --dirty --match "v[0-9]*" ran
        1.43 ± 0.09 times faster than git describe --match "v[0-9]*" && git update-index -q --refresh && git diff-index --name-only HEAD --

While the speedup doesn't really matter on Unix-based systems, where filesystem operations are typically fast, they do matter on Windows where the commands take a couple hundred milliseconds. A quick and dirty check on that system shows a speedup from ~800ms to ~400ms.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 GIT-VERSION-GEN | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

The GIT-VERSION-GEN script requires an input file containing formatting directives to be replaced as well as an output file that will get overwritten in case the file contents have changed. When computing the project version for Meson we don't want to have either though:

  - We only want to compute the version without anything else, but don't
    have an input file that would match that exact format. While we
    could of course introduce a new file just for that usecase, it feels
    suboptimal to add another file every time we want to have a slightly
    different format for versioned data.

  - The computed version needs to be read from stdout so that Meson can
    wire it up for the project.

Extend the script to handle both usecases by recognizing `--format=` as alternative to providing an input path and by writing to stdout in case no output file was given.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 GIT-VERSION-GEN | 44 +++++++++++++++++++++++++++++---------------
 1 file changed, 29 insertions(+), 15 deletions(-)

Patrick Steinhardt <ps@pks.im> writes:

I was considering to suggest to have GIT-VERSION-GEN accept --stdin for $INPUT and have it read from stdin instead of passing the format as an argument, but looking at the following commits I agree with your approach.

-- Toon

The Git version for Meson is currently wired up manually. It can thus grow (and alread has grown) stale quite easily, as having multiple sources of truth is never a good idea. This issue is mostly of cosmetic nature as we don't use the project version anywhere, and instead use the GIT-VERSION-GEN script to propagate the correct version into our build. But it is somewhat puzzling when `meson setup` announces to build an old Git release.

There are a couple of alternatives for how to solve this:

  - We can keep the version undefined, but this makes Meson output
    "undefined" for the version, as well.

  - We can use GIT-VERSION-GEN to generate the version for us. At the
    point of configuring the project we haven't yet figured out host
    details though, and thus we didn't yet set up the shell environment.
    While not an issue for Unix-based systems, this would be an issue in
    Windows, where the shell typically gets provided via Git for Windows
    and thus requires some special setup.

  - We can pull the default version out of GIT-VERSION-GEN and move it
    into its own file. This likely requires some adjustments for scripts
    that bump the version, but allows Meson to read the version from
    that file trivially.

Pick the second option and use GIT-VERSION-GEN as it gives us the most accurate version. In order to fix the bootstrapping issue on Windows systems we simply set the version to 'unknown' in case no shell was found. As the version is only of cosmetic value this isn't really much of an issue.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

Patrick Steinhardt <ps@pks.im> writes:

Small typo: alread -> already

We generate a couple of headers from our documentation. These headers are added to the libgit sources, but two of them aren't used by the library, but instead by our builtins. This can cause parallel builds to fail because the builtin object may be compiled before the header was generated.

Fix the issue by adding both "config-list.h" and "hook-list.h" to the list of builtin sources. While "command-list.h" is generated similarly, it is used by "help.c" and thus part of the libgit sources indeed.

Reported-by: Evan Martin <evan.martin@gmail.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

The Meson build system is able to wire up development environments. The intent is to make build artifacts of the project available. This is typically used to export e.g. paths to linkable libraries, which isn't all that interesting in our context given that we don't have an official library interface.

But what we can use this mechanism for is to expose the built Git executables as well as the build directory. This allows users to play around with the built Git version in the devenv, and allows them to execute our test scripts directly with the built distribution.

Wire up this feature, which can then be used via `meson devenv` in the build directory.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build | 8 ++++++++
 1 file changed, 8 insertions(+)

Meson knows to generate distribution archives via `meson dist`. In addition to generating the archive itself, this target also knows to compile and execute tests from that archive, which helps to ensure that the result is an adequate drop-in replacement for the versioned project.

While this already works as-is, one omission is that we don't propagate the commit that this is built from into the resulting archive. This can be fixed though by adding a distribution script that propagates the version into the "version" file, which GIT-VERSION-GEN knows to read if present.

Use GIT-VERSION-GEN to populate that file. As the script is executed in the build directory, not in the directory where we generate the archive, we have adapt it to honor the "MESON_DIST_ROOT" environment variable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 GIT-VERSION-GEN |  4 ++++
 meson.build     | 12 ++++++++++++
 2 files changed, 16 insertions(+)

Patrick Steinhardt <ps@pks.im> writes:

I failed to understand why you couldn't pass the absolute path of the output file to GIT-VERSION-GEN. So I looked at the previous version of this patch, and it seems you explain better over there.

I was testing things locally and tried this line for the last argument to the script:

  run_command(shell, '-c', 'echo $MESON_DIST_ROOT', capture: true, check: true).stdout().strip() / 'version',

And I think I understand it better now. Meson does not execute this when you run `meson dist`, but when it (re)generates it's build files. At that stage $MESON_DIST_ROOT is not set.

It's unfortunate we have to learn GIT-VERSION-GEN about the $MESON_DIST_ROOT environment variable, but I don't see any other way.

-- Toon

On Tue, Jan 21, 2025 at 01:37:23PM +0100, Toon Claes wrote:

Hm. Thinking about it a bit more there is an alternative:

    meson.add_dist_script(
      shell,
      '-c',
      '"$1" "$2" "$3" --format="@GIT_VERSION@" "$MESON_DIST_ROOT/version"',
      'GIT-VERSION-GEN',
      shell,
      meson.current_source_dir() / 'GIT-VERSION-GEN',
      meson.current_source_dir(),
    )

I think this is a much better solution as it doesn't require us to teach the script about `MESON_DIST_ROOT`.

Patrick

Meson does not yet know to build our fuzzers. Introduce a new build option "fuzzers" and wire up the fuzzers in case it is enabled. Adapt our CI jobs so that they build the fuzzers by default.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 ci/run-build-and-tests.sh |  3 ++-
 meson.build               |  4 ++++
 meson_options.txt         |  2 ++
 oss-fuzz/meson.build      | 20 ++++++++++++++++++++
 4 files changed, 28 insertions(+), 1 deletion(-)

The CSPRNG backend is not configurable in Meson and isn't quite discoverable, either. Make it configurable and add the actual backend used to the summary.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build       | 28 +++++++++++++++++++++-------
 meson_options.txt |  2 ++
 2 files changed, 23 insertions(+), 7 deletions(-)

The Visual Studio compiler defaults to C89 unless explicitly asked to use a different version of the C standard. We don't specify any C standard at all though in our Meson build, and consequently compiling Git fails:

    ...\git\git-compat-util.h(14): fatal error C1189: #error:  "Required C99 support is in a test phase.  Please see git-compat-util.h for more details."

Fix the issue by specifying the project's C standard. Funny enough, specifying C99 does not work because apparently, `__STDC_VERSION__` is not getting defined in that version at all. Instead, we have to specify C11 as the project's C standard, which is also done in our CMake build instructions.

We don't want to generally enforce C11 though, as our requiremets only state that a C99 compiler is required. In fact, we don't even require plain C99, but rather the GNU variant thereof.

Meson allows us to handle this case rather easily by specifying "gnu99,c11", which will cause it to fall back to C11 in case GNU C99 is unsupported. This feature has only been introduced with Meson 1.3.0 though, and we support 0.61.0 and newer. In case we use such an oldish version though we fall back to requiring GNU99 unconditionally. This means that Windows essentially requires Meson 1.3.0 and newer when using Visual Studio, but I doubt that this is ever going to be a real problem.

Tested-by: M Hickford <mirth.hickford@gmail.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 meson.build | 8 ++++++++
 1 file changed, 8 insertions(+)

Meson prints warnings in several cases, like for example when using a feature supported by the current version of Meson, but not yet supported by the minimum required version as declared by the project. These warnings will not cause the setup to fail by default, which makes it quite easy to miss them.

Improve this by passing `--fatal-meson-warnings` to `meson setup` so that our CI jobs will fail on warnings.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 ci/run-build-and-tests.sh | 1 +
 1 file changed, 1 insertion(+)

Patrick Steinhardt <ps@pks.im> writes:

Cool, this would have caught me trying to use `fs.copyfile`:

    $ meson setup build --fatal-meson-warnings

    # ... [snip] ...

    ../meson.build:1466: WARNING: Project targets '>=0.61.0' but uses feature introduced in '0.64.0': fs.copyfile.

    ../meson.build:1466:3: ERROR: Fatal warnings enabled, aborting

(I missed that warning)

-- Toon

Add a new job to GitHub Actions and GitLab CI that builds and tests Meson-based builds with Visual Studio.

A couple notes:

  - While the build job is mandatory, the test job is marked as "manual"
    on GitLab so that it doesn't run by default. We already have a bunch
    of Windows-based jobs, and the computational overhead that these
    cause is simply out of proportion to run the test suite twice.

    The same isn't true for GitHub as I could not find a way to make a
    subset of jobs manually triggered.

  - We disable Perl. This is because we pick up Perl from Git for
    Windows, which outputs different paths ("/c/" instead of "C:\") than
    what we expect in our tests.

  - We don't use the Git for Windows SDK. Instead, the build only
    depends on Visual Studio, Meson and Git for Windows. All the other
    dependencies like curl, pcre2 and zlib get pulled in and compiled
    automatically by Meson and thus do not have to be provided by the
    system.

  - We open-code "ci/run-test-slice.sh". This is because we only have
    direct access to PowerShell, so we manually implement the logic.
    There is an upstream pull request for the Meson build system [1] to
    implement test slicing in Meson directly.

  - We don't process test artifacts for failed CI jobs. This is done to
    keep down prerequisites to a minimum.

All tests are passing.

[1]: https://github.com/mesonbuild/meson/pull/14092

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 .github/workflows/main.yml | 52 ++++++++++++++++++++++++++++++++++++++++++++++
 .gitlab-ci.yml             | 38 +++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)

Patrick Steinhardt <ps@pks.im> writes:

Ahh, you only said "found the issue" and it was building suspense ;-)

It turns out that I wasn't affected while testing because my build infrastructure around "make" computes the version string by itself independently from what "make/GIT-VERSION-GEN" does, and stuffs it in the 'version' file, before running "make install". And that is why I didn't notice the breakage before Ramsay reported X-<.

This fix looks obviously good.

Thanks.