git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 5/8] ci: unify setup of some environment variables

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 30, 2023, 15:19 UTC
Message-ID
<ZT_JcNH79ncQhzPD@tanuki>
In-Reply-To
<87430c6c-91c0-4be1-b89d-bf442b3f018b@gmail.com>
On Mon, Oct 30, 2023 at 03:09:01PM +0000, Phillip Wood wrote:
Show 23 quoted lines
> Hi Patrick
> 
> On 30/10/2023 12:15, Patrick Steinhardt wrote:
> > Both GitHub Actions and Azue Pipelines set up the environment variables
> > GIT_TEST_OPTS, GIT_PROVE_OPTS and MAKEFLAGS. And while most values are
> > actually the same, the setup is completely duplicate. With the upcoming
> > support for GitLab CI this duplication would only extend even further.
> > 
> > Unify the setup of those environment variables so that only the uncommon
> > parts are separated. While at it, we also perform some additional small
> > improvements:
> > 
> >      - We use nproc instead of a hardcoded count of jobs for make and
> >        prove. This ensures that the number of concurrent processes adapts
> >        to the host automatically.
> 
> Sadly this makes the Windows and MacOS jobs fail on GitHub Actions as nproc
> is not installed[1]. Perhaps we could do
> 
> 	--jobs="$(nproc || echo 2)"
> 
> instead. (Maybe 2 is a bit low but the current value of 10 seems pretty high
> for the number of cores on the runners that we use)

Ugh, thanks. I'll update it to keep the hardcoded 10 jobs in place for now for the other pipelines. Trying to do too many things at once is only going to make it harder to get this landed, doubly so when you have no easy way to verify what you're doing :)

Show 12 quoted lines
> >      - We now always pass `--state=failed,slow,save` via GIT_PROVE_OPTS.
> >        It doesn't hurt on platforms where we don't persist the state, so
> >        this further reduces boilerplate.
> > 
> >      - When running on Windows systems we set `--no-chain-lint` and
> >        `--no-bin-wrappers`. Interestingly though, we did so _after_
> >        already having exported the respective environment variables.
> > >      - We stop using `export VAR=value` syntax, which is a Bashism. It's
> >        not quite worth it as we still use this syntax all over the place,
> >        but it doesn't hurt readability either.
> 
> I don't mind this change, but the 'export VAR=value' syntax is in POSIX[2]

I don't quite mind it, either. The reason why I chose to use it is that there was indeed a bug already with the order of exports and modification of the vars after the export. So with that in mind it made sense to me to adapt it accordingly.

Show 49 quoted lines
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> >   ci/lib.sh | 24 ++++++++++++++----------
> >   1 file changed, 14 insertions(+), 10 deletions(-)
> > 
> > diff --git a/ci/lib.sh b/ci/lib.sh
> > index 9ffdf743903..c7a716a6e3f 100755
> > --- a/ci/lib.sh
> > +++ b/ci/lib.sh
> > @@ -175,11 +175,7 @@ then
> >   	# among *all* phases)
> >   	cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME"
> > -	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
> > -	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
> > -	MAKEFLAGS="$MAKEFLAGS --jobs=10"
> > -	test windows_nt != "$CI_OS_NAME" ||
> > -	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
> > +	GIT_TEST_OPTS="--write-junit-xml"
> >   elif test true = "$GITHUB_ACTIONS"
> >   then
> >   	CI_TYPE=github-actions
> > @@ -198,17 +194,25 @@ then
> >   	cache_dir="$HOME/none"
> > -	export GIT_PROVE_OPTS="--timer --jobs 10"
> > -	export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
> > -	MAKEFLAGS="$MAKEFLAGS --jobs=10"
> > -	test windows != "$CI_OS_NAME" ||
> > -	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
> > +	GIT_TEST_OPTS="--github-workflow-markup"
> >   else
> >   	echo "Could not identify CI type" >&2
> >   	env >&2
> >   	exit 1
> >   fi
> > +MAKEFLAGS="$MAKEFLAGS --jobs=$(nproc)"
> > +GIT_PROVE_OPTS="--timer --jobs $(nproc) --state=failed,slow,save"
> > +
> > +GIT_TEST_OPTS="$GIT_TEST_OPTS --verbose-log -x"
> > +if test windows = "$CI_OS_NAME"
> > +then
> > +	GIT_TEST_OPTS="$GIT_TEST_OPTS --no-chain-lint --no-bin-wrappers"
> > +fi
> >
> > +export GIT_TEST_OPTS
> > +export GIT_PROVE_OPTS
> 
> I was wondering why we don't export MAKEFLAGS here but it is exported
> earlier on before we set it. Apart from the nproc issue this looks like a
> nice improvement

In fact, I also noticed later today that we adapt MAKEFLAGS at a later point again without reexporting it, so it's got the same issue. I'll adapt it in the same way.

Patrick
Previous: Phillip WoodNext: Dragan Simic
Message 51 of 101 in “ci: add GitLab CI definition”
  1. 0/5 ci: add GitLab CI definitionPatrick Steinhardt, Oct 26, 2023
  2. 1/5 ci: reorder definitions for grouping functionsPatrick Steinhardt, Oct 26, 2023
  3. Oswald BuddenhagenOct 26, 2023
  4. Patrick SteinhardtOct 27, 2023
  5. 2/5 ci: make grouping setup more genericPatrick Steinhardt, Oct 26, 2023
  6. 3/5 ci: group installation of Docker dependenciesPatrick Steinhardt, Oct 26, 2023
  7. Oswald BuddenhagenOct 26, 2023
  8. Patrick SteinhardtOct 27, 2023
  9. 4/5 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Oct 26, 2023
  10. Oswald BuddenhagenOct 26, 2023
  11. Christian CouderNov 3, 2023
  12. Patrick SteinhardtNov 6, 2023
  13. 5/5 ci: add support for GitLab CIPatrick Steinhardt, Oct 26, 2023
  14. Oswald BuddenhagenOct 26, 2023
  15. Patrick SteinhardtOct 27, 2023
  16. Phillip WoodOct 27, 2023
  17. Oswald BuddenhagenOct 27, 2023
  18. Phillip WoodOct 27, 2023
  19. Oswald BuddenhagenOct 27, 2023
  20. Phillip WoodOct 30, 2023
  21. Dragan SimicOct 30, 2023
  22. Oswald BuddenhagenOct 27, 2023
  23. Patrick SteinhardtOct 27, 2023
  24. 0/5 ci: add GitLab CI definitionPatrick Steinhardt, Oct 27, 2023
  25. 1/5 ci: reorder definitions for grouping functionsPatrick Steinhardt, Oct 27, 2023
  26. 2/5 ci: make grouping setup more genericPatrick Steinhardt, Oct 27, 2023
  27. 3/5 ci: group installation of Docker dependenciesPatrick Steinhardt, Oct 27, 2023
  28. 4/5 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Oct 27, 2023
  29. 5/5 ci: add support for GitLab CIPatrick Steinhardt, Oct 27, 2023
  30. Phillip WoodOct 27, 2023
  31. Patrick SteinhardtOct 27, 2023
  32. Patrick SteinhardtOct 27, 2023
  33. Phillip WoodOct 27, 2023
  34. Phillip WoodOct 29, 2023
  35. Patrick SteinhardtOct 30, 2023
  36. Phillip WoodOct 29, 2023
  37. Patrick SteinhardtOct 30, 2023
  38. Junio C HamanoOct 30, 2023
  39. Oswald BuddenhagenOct 27, 2023
  40. Phillip WoodOct 27, 2023
  41. Oswald BuddenhagenOct 27, 2023
  42. Jeff KingOct 31, 2023
  43. Junio C HamanoNov 1, 2023
  44. 0/8 ci: add GitLab CI definitionPatrick Steinhardt, Oct 30, 2023
  45. 1/8 ci: reorder definitions for grouping functionsPatrick Steinhardt, Oct 30, 2023
  46. 2/8 ci: make grouping setup more genericPatrick Steinhardt, Oct 30, 2023
  47. 3/8 ci: group installation of Docker dependenciesPatrick Steinhardt, Oct 30, 2023
  48. 4/8 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Oct 30, 2023
  49. 5/8 ci: unify setup of some environment variablesPatrick Steinhardt, Oct 30, 2023
  50. Phillip WoodOct 30, 2023
  51. Patrick SteinhardtOct 30, 2023
  52. Dragan SimicOct 30, 2023
  53. 6/8 ci: squelch warnings when testing with unusable Git repoPatrick Steinhardt, Oct 30, 2023
  54. 7/8 ci: install test dependencies for linux-muslPatrick Steinhardt, Oct 30, 2023
  55. Patrick SteinhardtOct 30, 2023
  56. Patrick SteinhardtOct 30, 2023
  57. Phillip WoodOct 30, 2023
  58. Patrick SteinhardtOct 30, 2023
  59. Phillip WoodOct 30, 2023
  60. 8/8 ci: add support for GitLab CIPatrick Steinhardt, Oct 30, 2023
  61. Taylor BlauOct 30, 2023
  62. Patrick SteinhardtOct 31, 2023
  63. Taylor BlauOct 31, 2023
  64. Junio C HamanoNov 1, 2023
  65. Patrick SteinhardtNov 1, 2023
  66. 0/8 ci: add GitLab CI definitionPatrick Steinhardt, Oct 31, 2023
  67. 1/8 ci: reorder definitions for grouping functionsPatrick Steinhardt, Oct 31, 2023
  68. 2/8 ci: make grouping setup more genericPatrick Steinhardt, Oct 31, 2023
  69. 3/8 ci: group installation of Docker dependenciesPatrick Steinhardt, Oct 31, 2023
  70. 4/8 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Oct 31, 2023
  71. 5/8 ci: unify setup of some environment variablesPatrick Steinhardt, Oct 31, 2023
  72. Victoria DyeOct 31, 2023
  73. Junio C HamanoNov 1, 2023
  74. Patrick SteinhardtNov 1, 2023
  75. 6/8 ci: squelch warnings when testing with unusable Git repoPatrick Steinhardt, Oct 31, 2023
  76. 7/8 ci: install test dependencies for linux-muslPatrick Steinhardt, Oct 31, 2023
  77. 8/8 ci: add support for GitLab CIPatrick Steinhardt, Oct 31, 2023
  78. Victoria DyeOct 31, 2023
  79. Patrick SteinhardtNov 1, 2023
  80. Victoria DyeOct 31, 2023
  81. Junio C HamanoNov 1, 2023
  82. Patrick SteinhardtNov 1, 2023
  83. 0/8 ci: add support for GitLab CIPatrick Steinhardt, Nov 1, 2023
  84. 1/8 ci: reorder definitions for grouping functionsPatrick Steinhardt, Nov 1, 2023
  85. 2/8 ci: make grouping setup more genericPatrick Steinhardt, Nov 1, 2023
  86. 3/8 ci: group installation of Docker dependenciesPatrick Steinhardt, Nov 1, 2023
  87. 4/8 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Nov 1, 2023
  88. 5/8 ci: unify setup of some environment variablesPatrick Steinhardt, Nov 1, 2023
  89. 6/8 ci: squelch warnings when testing with unusable Git repoPatrick Steinhardt, Nov 1, 2023
  90. 7/8 ci: install test dependencies for linux-muslPatrick Steinhardt, Nov 1, 2023
  91. 8/8 ci: add support for GitLab CIPatrick Steinhardt, Nov 1, 2023
  92. 0/8 ci: add GitLab CI definitionPatrick Steinhardt, Nov 9, 2023
  93. 1/8 ci: reorder definitions for grouping functionsPatrick Steinhardt, Nov 9, 2023
  94. 2/8 ci: make grouping setup more genericPatrick Steinhardt, Nov 9, 2023
  95. 3/8 ci: group installation of Docker dependenciesPatrick Steinhardt, Nov 9, 2023
  96. 4/8 ci: split out logic to set up failed test artifactsPatrick Steinhardt, Nov 9, 2023
  97. 5/8 ci: unify setup of some environment variablesPatrick Steinhardt, Nov 9, 2023
  98. 6/8 ci: squelch warnings when testing with unusable Git repoPatrick Steinhardt, Nov 9, 2023
  99. 7/8 ci: install test dependencies for linux-muslPatrick Steinhardt, Nov 9, 2023
  100. 8/8 ci: add support for GitLab CIPatrick Steinhardt, Nov 9, 2023
  101. Junio C HamanoNov 9, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.