threads / patch / 53103

patch, 3 partsadd travis job for linux with musl libc

Subject: [PATCH 0/3] add travis job for linux with musl libc

## tl;dr

42 messages between Mar 26, 2020 and Apr 10, 2020. Diffs are folded; open one to read it.

replies: 41people: 5as markdown or json

Đoàn Trần Công Danh· Mar 26, 2020, 07:35 UTC · lore

Recently, we've un-broken git for Linux with musl libc, and we have a serie to fix false negative with busybox shell utils.

There is a sample travis build for this serie applied on top of master: https://travis-ci.org/github/sgn/git/builds/667111988

And, after merging with my v4 to fix busybox false negative: https://travis-ci.org/github/sgn/git/builds/667112197

Đoàn Trần Công Danh (3):
  ci: libify logic for usage and checking CI_USER
  ci: refactor docker runner script
  travis: build and test on Linux with musl libc and busybox
 .travis.yml                                 | 10 +++++-
 azure-pipelines.yml                         |  4 +--
 ci/lib-docker.sh                            | 37 +++++++++++++++++++++
 ci/run-alpine-build.sh                      | 31 +++++++++++++++++
 ci/{run-linux32-docker.sh => run-docker.sh} | 20 +++++++----
 ci/run-linux32-build.sh                     | 35 +------------------
 6 files changed, 94 insertions(+), 43 deletions(-)
 create mode 100644 ci/lib-docker.sh
 create mode 100755 ci/run-alpine-build.sh
 rename ci/{run-linux32-docker.sh => run-docker.sh} (46%)
-- 
2.26.0.rc2.357.g1e1ba0441d
Đoàn Trần Công Danh· Mar 26, 2020, 07:35 UTC · re: Đoàn Trần Công Danh · lore

[PATCH 1/3] ci: libify logic for usage and checking CI_USER

This part of logic will be reused for alpine docker later.
Merge those logic into single chunk since they will be used together.

While we're at it, add a comment to tell people run with root inside podman container.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
This patch is viewed better with "git diff --color-moved"
 ci/lib-docker.sh        | 37 +++++++++++++++++++++++++++++++++++++
 ci/run-linux32-build.sh | 35 +----------------------------------
 2 files changed, 38 insertions(+), 34 deletions(-)
 create mode 100644 ci/lib-docker.sh
Show changes to 2 files +38 −34

ci/lib-docker.sh, ci/run-linux32-build.sh

diff --git a/ci/lib-docker.sh b/ci/lib-docker.sh
new file mode 100644
index 0000000000..ac155ace54
--- /dev/null
+++ b/ci/lib-docker.sh
@@ -0,0 +1,37 @@
+# Library of functions shared by all CI scripts run inside docker
+
+if test $# -ne 1 || test -z "$1"
+then
+	echo >&2 "usage: $0 <host-user-id>"
+	exit 1
+fi
+
+# If this script runs inside a docker container, then all commands are
+# usually executed as root. Consequently, the host user might not be
+# able to access the test output files.
+# If a non 0 host user id is given, then create a user "ci" with that
+# user id to make everything accessible to the host user.
+HOST_UID=$1
+if test $HOST_UID -eq 0
+then
+	# Just in case someone does want to run the test suite as root.
+	# or podman is used in place of docker
+	CI_USER=root
+else
+	CI_USER=ci
+	if test "$(id -u $CI_USER 2>/dev/null)" = $HOST_UID
+	then
+		echo "user '$CI_USER' already exists with the requested ID $HOST_UID"
+	else
+		useradd -u $HOST_UID $CI_USER
+	fi
+
+	# Due to a bug the test suite was run as root in the past, so
+	# a prove state file created back then is only accessible by
+	# root.  Now that bug is fixed, the test suite is run as a
+	# regular user, but the prove state file coming from Travis
+	# CI's cache might still be owned by root.
+	# Make sure that this user has rights to any cached files,
+	# including an existing prove state file.
+	test -n "$cache_dir" && chown -R $HOST_UID:$HOST_UID "$cache_dir"
+fi
diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh
index e3a193adbc..81296cdd19 100755
--- a/ci/run-linux32-build.sh
+++ b/ci/run-linux32-build.sh
@@ -8,11 +8,7 @@
 
 set -ex
 
-if test $# -ne 1 || test -z "$1"
-then
-	echo >&2 "usage: run-linux32-build.sh <host-user-id>"
-	exit 1
-fi
+. "${0%/*}/lib-docker.sh"
 
 # Update packages to the latest available versions
 linux32 --32bit i386 sh -c '
@@ -21,35 +17,6 @@ linux32 --32bit i386 sh -c '
 	libexpat-dev gettext python >/dev/null
 '
 
-# If this script runs inside a docker container, then all commands are
-# usually executed as root. Consequently, the host user might not be
-# able to access the test output files.
-# If a non 0 host user id is given, then create a user "ci" with that
-# user id to make everything accessible to the host user.
-HOST_UID=$1
-if test $HOST_UID -eq 0
-then
-	# Just in case someone does want to run the test suite as root.
-	CI_USER=root
-else
-	CI_USER=ci
-	if test "$(id -u $CI_USER 2>/dev/null)" = $HOST_UID
-	then
-		echo "user '$CI_USER' already exists with the requested ID $HOST_UID"
-	else
-		useradd -u $HOST_UID $CI_USER
-	fi
-
-	# Due to a bug the test suite was run as root in the past, so
-	# a prove state file created back then is only accessible by
-	# root.  Now that bug is fixed, the test suite is run as a
-	# regular user, but the prove state file coming from Travis
-	# CI's cache might still be owned by root.
-	# Make sure that this user has rights to any cached files,
-	# including an existing prove state file.
-	test -n "$cache_dir" && chown -R $HOST_UID:$HOST_UID "$cache_dir"
-fi
-
 # Build and test
 linux32 --32bit i386 su -m -l $CI_USER -c '
 	set -ex

-- 
2.26.0.rc2.357.g1e1ba0441d
Đoàn Trần Công Danh· Mar 26, 2020, 07:35 UTC · re: Đoàn Trần Công Danh · lore

[PATCH 2/3] ci: refactor docker runner script

We will support alpine check in docker later in this serie.
While we're at it, tell people to run as root in podman.
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 .travis.yml                                 |  2 +-
 azure-pipelines.yml                         |  4 ++--
 ci/{run-linux32-docker.sh => run-docker.sh} | 19 +++++++++++++------
 3 files changed, 16 insertions(+), 9 deletions(-)
 rename ci/{run-linux32-docker.sh => run-docker.sh} (48%)
Show changes to 4 files +16 −9

.travis.yml, azure-pipelines.yml, ci/run-linux32-docker.sh, ci/run-docker.sh

diff --git a/.travis.yml b/.travis.yml
index fc5730b085..32e80e2670 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -32,7 +32,7 @@ matrix:
       services:
         - docker
       before_install:
-      script: ci/run-linux32-docker.sh
+      script: ci/run-docker.sh linux32
     - env: jobname=StaticAnalysis
       os: linux
       compiler:
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 675c3a43c9..ef504ff29f 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -489,14 +489,14 @@ jobs:
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
 
        res=0
-       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-linux32-docker.sh || res=1
+       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-docker.sh linux32 || res=1
 
        sudo chmod a+r t/out/TEST-*.xml
        test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
 
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
        exit $res
-    displayName: 'ci/run-linux32-docker.sh'
+    displayName: 'ci/run-docker.sh linux32'
     env:
       GITFILESHAREPWD: $(gitfileshare.pwd)
   - task: PublishTestResults@2
diff --git a/ci/run-linux32-docker.sh b/ci/run-docker.sh
similarity index 48%
rename from ci/run-linux32-docker.sh
rename to ci/run-docker.sh
index 751acfcf8a..c8dff9d41a 100755
--- a/ci/run-linux32-docker.sh
+++ b/ci/run-docker.sh
@@ -1,15 +1,22 @@
 #!/bin/sh
 #
-# Download and run Docker image to build and test 32-bit Git
+# Download and run Docker image to build and test git
 #
 
 . ${0%/*}/lib.sh
 
-docker pull daald/ubuntu32:xenial
+CI_TARGET=${1:-linux32}
+case "$CI_TARGET" in
+linux32) CI_CONTAINER="daald/ubuntu32:xenial" ;;
+*)       exit 1 ;;
+esac
+
+docker pull "$CI_CONTAINER"
 
 # Use the following command to debug the docker build locally:
-# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/bash daald/ubuntu32:xenial
-# root@container:/# /usr/src/git/ci/run-linux32-build.sh <host-user-id>
+# <host-user-id> must be 0 if podman is used in place of docker
+# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/sh "$CI_CONTAINER"
+# root@container:/# /usr/src/git/ci/run-$CI_TARGET-build.sh <host-user-id>
 
 container_cache_dir=/tmp/travis-cache
 
@@ -23,8 +30,8 @@ docker run \
 	--env cache_dir="$container_cache_dir" \
 	--volume "${PWD}:/usr/src/git" \
 	--volume "$cache_dir:$container_cache_dir" \
-	daald/ubuntu32:xenial \
-	/usr/src/git/ci/run-linux32-build.sh $(id -u $USER)
+	"$CI_CONTAINER" \
+	"/usr/src/git/ci/run-$CI_TARGET-build.sh" $(id -u $USER)
 
 check_unignored_build_artifacts
 

-- 
2.26.0.rc2.357.g1e1ba0441d
Eric Sunshine· Mar 26, 2020, 16:06 UTC · re: Đoàn Trần Công Danh · lore

Re: [PATCH 2/3] ci: refactor docker runner script

On Thu, Mar 26, 2020 at 3:35 AM Đoàn Trần Công Danh <congdanhqx@gmail.com> wrote:

> We will support alpine check in docker later in this serie.
s/serie/series/
> While we're at it, tell people to run as root in podman.
>
> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
SZEDER Gábor· Mar 28, 2020, 17:53 UTC · re: Đoàn Trần Công Danh · lore

Re: [PATCH 2/3] ci: refactor docker runner script

On Thu, Mar 26, 2020 at 02:35:18PM +0700, Đoàn Trần Công Danh wrote:
> We will support alpine check in docker later in this serie.
> 
> While we're at it, tell people to run as root in podman.

Why tell that to people? Please clarify what podman is and why should we care.

Show 18 quoted lines
> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> ---
>  .travis.yml                                 |  2 +-
>  azure-pipelines.yml                         |  4 ++--
>  ci/{run-linux32-docker.sh => run-docker.sh} | 19 +++++++++++++------
>  3 files changed, 16 insertions(+), 9 deletions(-)
>  rename ci/{run-linux32-docker.sh => run-docker.sh} (48%)
> 
> diff --git a/.travis.yml b/.travis.yml
> index fc5730b085..32e80e2670 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -32,7 +32,7 @@ matrix:
>        services:
>          - docker
>        before_install:
> -      script: ci/run-linux32-docker.sh
> +      script: ci/run-docker.sh linux32

The name of the 'Linux32' build job starts with a capital 'L'; please be consistent with that.

Show 36 quoted lines
>      - env: jobname=StaticAnalysis
>        os: linux
>        compiler:
> diff --git a/azure-pipelines.yml b/azure-pipelines.yml
> index 675c3a43c9..ef504ff29f 100644
> --- a/azure-pipelines.yml
> +++ b/azure-pipelines.yml
> @@ -489,14 +489,14 @@ jobs:
>         test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
>  
>         res=0
> -       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-linux32-docker.sh || res=1
> +       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-docker.sh linux32 || res=1
>  
>         sudo chmod a+r t/out/TEST-*.xml
>         test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
>  
>         test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
>         exit $res
> -    displayName: 'ci/run-linux32-docker.sh'
> +    displayName: 'ci/run-docker.sh linux32'
>      env:
>        GITFILESHAREPWD: $(gitfileshare.pwd)
>    - task: PublishTestResults@2
> diff --git a/ci/run-linux32-docker.sh b/ci/run-docker.sh
> similarity index 48%
> rename from ci/run-linux32-docker.sh
> rename to ci/run-docker.sh
> index 751acfcf8a..c8dff9d41a 100755
> --- a/ci/run-linux32-docker.sh
> +++ b/ci/run-docker.sh
> @@ -1,15 +1,22 @@
>  #!/bin/sh
>  #
> -# Download and run Docker image to build and test 32-bit Git
> +# Download and run Docker image to build and test git
s/git/Git/
Show 36 quoted lines
>  #
>  
>  . ${0%/*}/lib.sh
>  
> -docker pull daald/ubuntu32:xenial
> +CI_TARGET=${1:-linux32}
> +case "$CI_TARGET" in
> +linux32) CI_CONTAINER="daald/ubuntu32:xenial" ;;
> +*)       exit 1 ;;
> +esac
> +
> +docker pull "$CI_CONTAINER"
>  
>  # Use the following command to debug the docker build locally:
> -# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/bash daald/ubuntu32:xenial
> -# root@container:/# /usr/src/git/ci/run-linux32-build.sh <host-user-id>
> +# <host-user-id> must be 0 if podman is used in place of docker
> +# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/sh "$CI_CONTAINER"
> +# root@container:/# /usr/src/git/ci/run-$CI_TARGET-build.sh <host-user-id>
>  
>  container_cache_dir=/tmp/travis-cache
>  
> @@ -23,8 +30,8 @@ docker run \
>  	--env cache_dir="$container_cache_dir" \
>  	--volume "${PWD}:/usr/src/git" \
>  	--volume "$cache_dir:$container_cache_dir" \
> -	daald/ubuntu32:xenial \
> -	/usr/src/git/ci/run-linux32-build.sh $(id -u $USER)
> +	"$CI_CONTAINER" \
> +	"/usr/src/git/ci/run-$CI_TARGET-build.sh" $(id -u $USER)
>  
>  check_unignored_build_artifacts
>  
> -- 
> 2.26.0.rc2.357.g1e1ba0441d
>
Danh Doan· Mar 29, 2020, 06:36 UTC · re: SZEDER Gábor · lore

Re: [PATCH 2/3] ci: refactor docker runner script

On 2020-03-28 18:53:29+0100, SZEDER Gábor <szeder.dev@gmail.com> wrote:
Show 7 quoted lines
> On Thu, Mar 26, 2020 at 02:35:18PM +0700, Đoàn Trần Công Danh wrote:
> > We will support alpine check in docker later in this serie.
> > 
> > While we're at it, tell people to run as root in podman.
> 
> Why tell that to people?  Please clarify what podman is and why should
> we care.

podman is a docker drop-in placement. I use it instead of docker to develop this series.

docker requires a service to be run and user in docker system groups, podman requires neither.

root user in podman is mapped to host user. I run into trouble when develop this series in my local machine.

Show 21 quoted lines
> > Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> > ---
> >  .travis.yml                                 |  2 +-
> >  azure-pipelines.yml                         |  4 ++--
> >  ci/{run-linux32-docker.sh => run-docker.sh} | 19 +++++++++++++------
> >  3 files changed, 16 insertions(+), 9 deletions(-)
> >  rename ci/{run-linux32-docker.sh => run-docker.sh} (48%)
> > 
> > diff --git a/.travis.yml b/.travis.yml
> > index fc5730b085..32e80e2670 100644
> > --- a/.travis.yml
> > +++ b/.travis.yml
> > @@ -32,7 +32,7 @@ matrix:
> >        services:
> >          - docker
> >        before_install:
> > -      script: ci/run-linux32-docker.sh
> > +      script: ci/run-docker.sh linux32
> 
> The name of the 'Linux32' build job starts with a capital 'L'; please
> be consistent with that.

the old name of the script is run-linux32-docker, I think it's better to rename the job to all lowercase. All other jobs, except Documentation and static analysis are in lowercase.

Show 39 quoted lines
> 
> >      - env: jobname=StaticAnalysis
> >        os: linux
> >        compiler:
> > diff --git a/azure-pipelines.yml b/azure-pipelines.yml
> > index 675c3a43c9..ef504ff29f 100644
> > --- a/azure-pipelines.yml
> > +++ b/azure-pipelines.yml
> > @@ -489,14 +489,14 @@ jobs:
> >         test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
> >  
> >         res=0
> > -       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-linux32-docker.sh || res=1
> > +       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-docker.sh linux32 || res=1
> >  
> >         sudo chmod a+r t/out/TEST-*.xml
> >         test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
> >  
> >         test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
> >         exit $res
> > -    displayName: 'ci/run-linux32-docker.sh'
> > +    displayName: 'ci/run-docker.sh linux32'
> >      env:
> >        GITFILESHAREPWD: $(gitfileshare.pwd)
> >    - task: PublishTestResults@2
> > diff --git a/ci/run-linux32-docker.sh b/ci/run-docker.sh
> > similarity index 48%
> > rename from ci/run-linux32-docker.sh
> > rename to ci/run-docker.sh
> > index 751acfcf8a..c8dff9d41a 100755
> > --- a/ci/run-linux32-docker.sh
> > +++ b/ci/run-docker.sh
> > @@ -1,15 +1,22 @@
> >  #!/bin/sh
> >  #
> > -# Download and run Docker image to build and test 32-bit Git
> > +# Download and run Docker image to build and test git
> 
> s/git/Git/
Will change
-- 
Danh
Đoàn Trần Công Danh· Mar 26, 2020, 07:35 UTC · re: Đoàn Trần Công Danh · lore

[PATCH 3/3] travis: build and test on Linux with musl libc and busybox

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 .travis.yml            |  8 ++++++++
 ci/run-alpine-build.sh | 31 +++++++++++++++++++++++++++++++
 ci/run-docker.sh       |  1 +
 3 files changed, 40 insertions(+)
 create mode 100755 ci/run-alpine-build.sh
Show changes to 3 files +40 −0

.travis.yml, ci/run-alpine-build.sh, ci/run-docker.sh

diff --git a/.travis.yml b/.travis.yml
index 32e80e2670..a2927dd120 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -33,6 +33,14 @@ matrix:
         - docker
       before_install:
       script: ci/run-docker.sh linux32
+    - env: jobname=linux-musl-busybox
+      os: linux
+      compiler:
+      addons:
+      services:
+        - docker
+      before_install:
+      script: ci/run-docker.sh alpine
     - env: jobname=StaticAnalysis
       os: linux
       compiler:
diff --git a/ci/run-alpine-build.sh b/ci/run-alpine-build.sh
new file mode 100755
index 0000000000..c83df536e4
--- /dev/null
+++ b/ci/run-alpine-build.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+#
+# Build and test Git in Alpine Linux
+#
+# Usage:
+#   run-alpine-build.sh <host-user-id>
+#
+
+set -ex
+
+useradd () {
+	adduser -D "$@"
+}
+
+. "${0%/*}/lib-docker.sh"
+
+# Update packages to the latest available versions
+apk add --update autoconf build-base curl-dev openssl-dev expat-dev \
+	gettext pcre2-dev python3 musl-libintl >/dev/null
+
+# Build and test
+su -m -l $CI_USER -c '
+	set -ex
+	cd /usr/src/git
+	test -n "$cache_dir" && ln -s "$cache_dir/.prove" t/.prove
+	autoconf
+	echo "PYTHON_PATH=/usr/bin/python3" >config.mak
+	./configure --with-libpcre
+	make
+	make test
+'
diff --git a/ci/run-docker.sh b/ci/run-docker.sh
index c8dff9d41a..47affcd6d3 100755
--- a/ci/run-docker.sh
+++ b/ci/run-docker.sh
@@ -8,6 +8,7 @@
 CI_TARGET=${1:-linux32}
 case "$CI_TARGET" in
 linux32) CI_CONTAINER="daald/ubuntu32:xenial" ;;
+alpine)  CI_CONTAINER="alpine" ;;
 *)       exit 1 ;;
 esac
 

-- 
2.26.0.rc2.357.g1e1ba0441d
Junio C Hamano· Mar 29, 2020, 05:49 UTC · re: Đoàn Trần Công Danh · lore

Re: [PATCH 0/3] add travis job for linux with musl libc

Đoàn Trần Công Danh  <congdanhqx@gmail.com> writes:
Show 8 quoted lines
> Recently, we've un-broken git for Linux with musl libc,
> and we have a serie to fix false negative with busybox shell utils.
>
> There is a sample travis build for this serie applied on top of master:
> https://travis-ci.org/github/sgn/git/builds/667111988
>
> And, after merging with my v4 to fix busybox false negative:
> https://travis-ci.org/github/sgn/git/builds/667112197

I have this topic near the tip of 'pu' and it seems to be finding issues not in your build with 'master'. I'd expect that we'll be seeing patches to various parts of the system from you or others with musl libc to correct them, after which we can merge the topic down.

Thanks.
Đoàn Trần Công Danh· Mar 29, 2020, 10:12 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v2 0/4] Travis + Azure jobs for linux with musl libc

Recently, we've un-broken git for Linux with musl libc, and we have a serie to fix false negative with busybox shell utils.

Add a CI job on Travis and Azure to make sure we won't break it again.

There is a sample travis build for this serie applied on top of: jt/rebase-allow-duplicate https://travis-ci.org/github/sgn/git/builds/668300819

And, after merging with junio's pu to fix busybox false negative: https://travis-ci.org/github/sgn/git/builds/668300919 https://dev.azure.com/git/git/_build/results?buildId=1910&view=results

Change from v1:
- fix spelling
- run-docker.sh: use "jobname" environment variable instead of passing argument
- add linux-musl job on Azure
- Add 4th patch for jt/rebase-allow-duplicate (feel free to squash into
 jt/rebase-allow-duplicate)

The first 3 patches could be applied on top of master, but the last patch needs to be applied on top of jt/rebase-allow-duplicate

Đoàn Trần Công Danh (4):
  ci: libify logic for usage and checking CI_USER
  ci: refactor docker runner script
  travis: build and test on Linux with musl libc and busybox
  t3402: use POSIX compliant regex(7)
 .travis.yml                                 | 10 +++++-
 azure-pipelines.yml                         | 39 +++++++++++++++++++--
 ci/lib-docker.sh                            | 37 +++++++++++++++++++
 ci/run-alpine-build.sh                      | 31 ++++++++++++++++
 ci/{run-linux32-docker.sh => run-docker.sh} | 26 ++++++++++----
 ci/run-linux32-build.sh                     | 35 +-----------------
 t/t3402-rebase-merge.sh                     |  8 ++---
 7 files changed, 139 insertions(+), 47 deletions(-)
 create mode 100644 ci/lib-docker.sh
 create mode 100755 ci/run-alpine-build.sh
 rename ci/{run-linux32-docker.sh => run-docker.sh} (46%)
Range-diff against v1:
1:  f23f2a563a = 1:  1ec7c2024d ci: libify logic for usage and checking CI_USER
2:  6fd1370678 ! 2:  140e0ef390 ci: refactor docker runner script
    @@ Metadata
      ## Commit message ##
         ci: refactor docker runner script
     
    -    We will support alpine check in docker later in this serie.
    +    We will support alpine check in docker later in this series.
     
         While we're at it, tell people to run as root in podman.
     
    @@ .travis.yml: matrix:
              - docker
            before_install:
     -      script: ci/run-linux32-docker.sh
    -+      script: ci/run-docker.sh linux32
    ++      script: ci/run-docker.sh
          - env: jobname=StaticAnalysis
            os: linux
            compiler:
    @@ azure-pipelines.yml: jobs:
      
             res=0
     -       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-linux32-docker.sh || res=1
    -+       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-docker.sh linux32 || res=1
    ++       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" jobname=Linux32 bash -lxc ci/run-docker.sh || res=1
      
             sudo chmod a+r t/out/TEST-*.xml
             test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
    @@ azure-pipelines.yml: jobs:
             test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
             exit $res
     -    displayName: 'ci/run-linux32-docker.sh'
    -+    displayName: 'ci/run-docker.sh linux32'
    ++    displayName: 'jobname=Linux32 ci/run-docker.sh'
          env:
            GITFILESHAREPWD: $(gitfileshare.pwd)
        - task: PublishTestResults@2
    @@ ci/run-docker.sh (new)
     @@
     +#!/bin/sh
     +#
    -+# Download and run Docker image to build and test git
    ++# Download and run Docker image to build and test Git
     +#
     +
     +. ${0%/*}/lib.sh
     +
    -+CI_TARGET=${1:-linux32}
    -+case "$CI_TARGET" in
    -+linux32) CI_CONTAINER="daald/ubuntu32:xenial" ;;
    -+*)       exit 1 ;;
    ++case "$jobname" in
    ++Linux32)
    ++	CI_TARGET=linux32
    ++	CI_CONTAINER="daald/ubuntu32:xenial"
    ++	;;
    ++*)
    ++	exit 1 ;;
     +esac
     +
     +docker pull "$CI_CONTAINER"
3:  2f68e65fb7 ! 3:  6cf6400f2e travis: build and test on Linux with musl libc and busybox
    @@ .travis.yml
     @@ .travis.yml: matrix:
              - docker
            before_install:
    -       script: ci/run-docker.sh linux32
    -+    - env: jobname=linux-musl-busybox
    +       script: ci/run-docker.sh
    ++    - env: jobname=linux-musl
     +      os: linux
     +      compiler:
     +      addons:
     +      services:
     +        - docker
     +      before_install:
    -+      script: ci/run-docker.sh alpine
    ++      script: ci/run-docker.sh
          - env: jobname=StaticAnalysis
            os: linux
            compiler:
     
    + ## azure-pipelines.yml ##
    +@@ azure-pipelines.yml: jobs:
    +       PathtoPublish: t/failed-test-artifacts
    +       ArtifactName: failed-test-artifacts
    + 
    ++- job: linux_musl
    ++  displayName: linux-musl
    ++  condition: succeeded()
    ++  pool:
    ++    vmImage: ubuntu-latest
    ++  steps:
    ++  - bash: |
    ++       test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
    ++
    ++       res=0
    ++       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" jobname=linux-musl bash -lxc ci/run-docker.sh || res=1
    ++
    ++       sudo chmod a+r t/out/TEST-*.xml
    ++       test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
    ++
    ++       test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
    ++       exit $res
    ++    displayName: 'jobname=linux-musl ci/run-docker.sh'
    ++    env:
    ++      GITFILESHAREPWD: $(gitfileshare.pwd)
    ++  - task: PublishTestResults@2
    ++    displayName: 'Publish Test Results **/TEST-*.xml'
    ++    inputs:
    ++      mergeTestResults: true
    ++      testRunTitle: 'musl'
    ++      platform: Linux
    ++      publishRunAttachments: false
    ++    condition: succeededOrFailed()
    ++  - task: PublishBuildArtifacts@1
    ++    displayName: 'Publish trash directories of failed tests'
    ++    condition: failed()
    ++    inputs:
    ++      PathtoPublish: t/failed-test-artifacts
    ++      ArtifactName: failed-test-artifacts
    ++
    + - job: static_analysis
    +   displayName: StaticAnalysis
    +   condition: succeeded()
    +
      ## ci/run-alpine-build.sh (new) ##
     @@
     +#!/bin/sh
    @@ ci/run-alpine-build.sh (new)
     +'
     
      ## ci/run-docker.sh ##
    -@@
    - CI_TARGET=${1:-linux32}
    - case "$CI_TARGET" in
    - linux32) CI_CONTAINER="daald/ubuntu32:xenial" ;;
    -+alpine)  CI_CONTAINER="alpine" ;;
    - *)       exit 1 ;;
    +@@ ci/run-docker.sh: Linux32)
    + 	CI_TARGET=linux32
    + 	CI_CONTAINER="daald/ubuntu32:xenial"
    + 	;;
    ++linux-musl)
    ++	CI_TARGET=alpine
    ++	CI_CONTAINER=alpine
    ++	;;
    + *)
    + 	exit 1 ;;
      esac
    - 
-:  ---------- > 4:  a4eacb4362 t3402: use POSIX compliant regex(7)
-- 
2.26.0.302.g234993491e
Đoàn Trần Công Danh· Mar 29, 2020, 10:12 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v2 1/4] ci: libify logic for usage and checking CI_USER

This part of logic will be reused for alpine docker later.
Merge those logic into single chunk since they will be used together.

While we're at it, add a comment to tell people run with root inside podman container.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 ci/lib-docker.sh        | 37 +++++++++++++++++++++++++++++++++++++
 ci/run-linux32-build.sh | 35 +----------------------------------
 2 files changed, 38 insertions(+), 34 deletions(-)
 create mode 100644 ci/lib-docker.sh
Show changes to 2 files +38 −34

ci/lib-docker.sh, ci/run-linux32-build.sh

diff --git a/ci/lib-docker.sh b/ci/lib-docker.sh
new file mode 100644
index 0000000000..ac155ace54
--- /dev/null
+++ b/ci/lib-docker.sh
@@ -0,0 +1,37 @@
+# Library of functions shared by all CI scripts run inside docker
+
+if test $# -ne 1 || test -z "$1"
+then
+	echo >&2 "usage: $0 <host-user-id>"
+	exit 1
+fi
+
+# If this script runs inside a docker container, then all commands are
+# usually executed as root. Consequently, the host user might not be
+# able to access the test output files.
+# If a non 0 host user id is given, then create a user "ci" with that
+# user id to make everything accessible to the host user.
+HOST_UID=$1
+if test $HOST_UID -eq 0
+then
+	# Just in case someone does want to run the test suite as root.
+	# or podman is used in place of docker
+	CI_USER=root
+else
+	CI_USER=ci
+	if test "$(id -u $CI_USER 2>/dev/null)" = $HOST_UID
+	then
+		echo "user '$CI_USER' already exists with the requested ID $HOST_UID"
+	else
+		useradd -u $HOST_UID $CI_USER
+	fi
+
+	# Due to a bug the test suite was run as root in the past, so
+	# a prove state file created back then is only accessible by
+	# root.  Now that bug is fixed, the test suite is run as a
+	# regular user, but the prove state file coming from Travis
+	# CI's cache might still be owned by root.
+	# Make sure that this user has rights to any cached files,
+	# including an existing prove state file.
+	test -n "$cache_dir" && chown -R $HOST_UID:$HOST_UID "$cache_dir"
+fi
diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh
index e3a193adbc..81296cdd19 100755
--- a/ci/run-linux32-build.sh
+++ b/ci/run-linux32-build.sh
@@ -8,11 +8,7 @@
 
 set -ex
 
-if test $# -ne 1 || test -z "$1"
-then
-	echo >&2 "usage: run-linux32-build.sh <host-user-id>"
-	exit 1
-fi
+. "${0%/*}/lib-docker.sh"
 
 # Update packages to the latest available versions
 linux32 --32bit i386 sh -c '
@@ -21,35 +17,6 @@ linux32 --32bit i386 sh -c '
 	libexpat-dev gettext python >/dev/null
 '
 
-# If this script runs inside a docker container, then all commands are
-# usually executed as root. Consequently, the host user might not be
-# able to access the test output files.
-# If a non 0 host user id is given, then create a user "ci" with that
-# user id to make everything accessible to the host user.
-HOST_UID=$1
-if test $HOST_UID -eq 0
-then
-	# Just in case someone does want to run the test suite as root.
-	CI_USER=root
-else
-	CI_USER=ci
-	if test "$(id -u $CI_USER 2>/dev/null)" = $HOST_UID
-	then
-		echo "user '$CI_USER' already exists with the requested ID $HOST_UID"
-	else
-		useradd -u $HOST_UID $CI_USER
-	fi
-
-	# Due to a bug the test suite was run as root in the past, so
-	# a prove state file created back then is only accessible by
-	# root.  Now that bug is fixed, the test suite is run as a
-	# regular user, but the prove state file coming from Travis
-	# CI's cache might still be owned by root.
-	# Make sure that this user has rights to any cached files,
-	# including an existing prove state file.
-	test -n "$cache_dir" && chown -R $HOST_UID:$HOST_UID "$cache_dir"
-fi
-
 # Build and test
 linux32 --32bit i386 su -m -l $CI_USER -c '
 	set -ex

-- 
2.26.0.302.g234993491e
Đoàn Trần Công Danh· Mar 29, 2020, 10:12 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v2 2/4] ci: refactor docker runner script

We will support alpine check in docker later in this series.
While we're at it, tell people to run as root in podman.
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 .travis.yml                                 |  2 +-
 azure-pipelines.yml                         |  4 ++--
 ci/{run-linux32-docker.sh => run-docker.sh} | 22 +++++++++++++++------
 3 files changed, 19 insertions(+), 9 deletions(-)
 rename ci/{run-linux32-docker.sh => run-docker.sh} (48%)
Show changes to 4 files +19 −9

.travis.yml, azure-pipelines.yml, ci/run-linux32-docker.sh, ci/run-docker.sh

diff --git a/.travis.yml b/.travis.yml
index fc5730b085..069aeeff3c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -32,7 +32,7 @@ matrix:
       services:
         - docker
       before_install:
-      script: ci/run-linux32-docker.sh
+      script: ci/run-docker.sh
     - env: jobname=StaticAnalysis
       os: linux
       compiler:
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index af2a5ea484..f6dcc35ad4 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -478,14 +478,14 @@ jobs:
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
 
        res=0
-       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-linux32-docker.sh || res=1
+       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" jobname=Linux32 bash -lxc ci/run-docker.sh || res=1
 
        sudo chmod a+r t/out/TEST-*.xml
        test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
 
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
        exit $res
-    displayName: 'ci/run-linux32-docker.sh'
+    displayName: 'jobname=Linux32 ci/run-docker.sh'
     env:
       GITFILESHAREPWD: $(gitfileshare.pwd)
   - task: PublishTestResults@2
diff --git a/ci/run-linux32-docker.sh b/ci/run-docker.sh
similarity index 48%
rename from ci/run-linux32-docker.sh
rename to ci/run-docker.sh
index 751acfcf8a..be698817cb 100755
--- a/ci/run-linux32-docker.sh
+++ b/ci/run-docker.sh
@@ -1,15 +1,25 @@
 #!/bin/sh
 #
-# Download and run Docker image to build and test 32-bit Git
+# Download and run Docker image to build and test Git
 #
 
 . ${0%/*}/lib.sh
 
-docker pull daald/ubuntu32:xenial
+case "$jobname" in
+Linux32)
+	CI_TARGET=linux32
+	CI_CONTAINER="daald/ubuntu32:xenial"
+	;;
+*)
+	exit 1 ;;
+esac
+
+docker pull "$CI_CONTAINER"
 
 # Use the following command to debug the docker build locally:
-# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/bash daald/ubuntu32:xenial
-# root@container:/# /usr/src/git/ci/run-linux32-build.sh <host-user-id>
+# <host-user-id> must be 0 if podman is used in place of docker
+# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/sh "$CI_CONTAINER"
+# root@container:/# /usr/src/git/ci/run-$CI_TARGET-build.sh <host-user-id>
 
 container_cache_dir=/tmp/travis-cache
 
@@ -23,8 +33,8 @@ docker run \
 	--env cache_dir="$container_cache_dir" \
 	--volume "${PWD}:/usr/src/git" \
 	--volume "$cache_dir:$container_cache_dir" \
-	daald/ubuntu32:xenial \
-	/usr/src/git/ci/run-linux32-build.sh $(id -u $USER)
+	"$CI_CONTAINER" \
+	"/usr/src/git/ci/run-$CI_TARGET-build.sh" $(id -u $USER)
 
 check_unignored_build_artifacts
 

-- 
2.26.0.302.g234993491e
SZEDER Gábor· Apr 1, 2020, 21:51 UTC · re: Đoàn Trần Công Danh · lore

Re: [PATCH v2 2/4] ci: refactor docker runner script

On Sun, Mar 29, 2020 at 05:12:30PM +0700, Đoàn Trần Công Danh wrote:
> We will support alpine check in docker later in this series.
> 
> While we're at it, tell people to run as root in podman.
Ok, I'll try to be more specific :)

Please clarify *in the commit message* why we should tell this to people, i.e. what podman is and why should we care.

Đoàn Trần Công Danh· Mar 29, 2020, 10:12 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v2 3/4] travis: build and test on Linux with musl libc and busybox

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 .travis.yml            |  8 ++++++++
 azure-pipelines.yml    | 35 +++++++++++++++++++++++++++++++++++
 ci/run-alpine-build.sh | 31 +++++++++++++++++++++++++++++++
 ci/run-docker.sh       |  4 ++++
 4 files changed, 78 insertions(+)
 create mode 100755 ci/run-alpine-build.sh
Show changes to 4 files +78 −0

.travis.yml, azure-pipelines.yml, ci/run-alpine-build.sh, ci/run-docker.sh

diff --git a/.travis.yml b/.travis.yml
index 069aeeff3c..0cfc3c3428 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -33,6 +33,14 @@ matrix:
         - docker
       before_install:
       script: ci/run-docker.sh
+    - env: jobname=linux-musl
+      os: linux
+      compiler:
+      addons:
+      services:
+        - docker
+      before_install:
+      script: ci/run-docker.sh
     - env: jobname=StaticAnalysis
       os: linux
       compiler:
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index f6dcc35ad4..615289167b 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -503,6 +503,41 @@ jobs:
       PathtoPublish: t/failed-test-artifacts
       ArtifactName: failed-test-artifacts
 
+- job: linux_musl
+  displayName: linux-musl
+  condition: succeeded()
+  pool:
+    vmImage: ubuntu-latest
+  steps:
+  - bash: |
+       test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
+
+       res=0
+       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" jobname=linux-musl bash -lxc ci/run-docker.sh || res=1
+
+       sudo chmod a+r t/out/TEST-*.xml
+       test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
+
+       test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
+       exit $res
+    displayName: 'jobname=linux-musl ci/run-docker.sh'
+    env:
+      GITFILESHAREPWD: $(gitfileshare.pwd)
+  - task: PublishTestResults@2
+    displayName: 'Publish Test Results **/TEST-*.xml'
+    inputs:
+      mergeTestResults: true
+      testRunTitle: 'musl'
+      platform: Linux
+      publishRunAttachments: false
+    condition: succeededOrFailed()
+  - task: PublishBuildArtifacts@1
+    displayName: 'Publish trash directories of failed tests'
+    condition: failed()
+    inputs:
+      PathtoPublish: t/failed-test-artifacts
+      ArtifactName: failed-test-artifacts
+
 - job: static_analysis
   displayName: StaticAnalysis
   condition: succeeded()
diff --git a/ci/run-alpine-build.sh b/ci/run-alpine-build.sh
new file mode 100755
index 0000000000..c83df536e4
--- /dev/null
+++ b/ci/run-alpine-build.sh
@@ -0,0 +1,31 @@
+#!/bin/sh
+#
+# Build and test Git in Alpine Linux
+#
+# Usage:
+#   run-alpine-build.sh <host-user-id>
+#
+
+set -ex
+
+useradd () {
+	adduser -D "$@"
+}
+
+. "${0%/*}/lib-docker.sh"
+
+# Update packages to the latest available versions
+apk add --update autoconf build-base curl-dev openssl-dev expat-dev \
+	gettext pcre2-dev python3 musl-libintl >/dev/null
+
+# Build and test
+su -m -l $CI_USER -c '
+	set -ex
+	cd /usr/src/git
+	test -n "$cache_dir" && ln -s "$cache_dir/.prove" t/.prove
+	autoconf
+	echo "PYTHON_PATH=/usr/bin/python3" >config.mak
+	./configure --with-libpcre
+	make
+	make test
+'
diff --git a/ci/run-docker.sh b/ci/run-docker.sh
index be698817cb..f203db03cf 100755
--- a/ci/run-docker.sh
+++ b/ci/run-docker.sh
@@ -10,6 +10,10 @@ Linux32)
 	CI_TARGET=linux32
 	CI_CONTAINER="daald/ubuntu32:xenial"
 	;;
+linux-musl)
+	CI_TARGET=alpine
+	CI_CONTAINER=alpine
+	;;
 *)
 	exit 1 ;;
 esac

-- 
2.26.0.302.g234993491e
SZEDER Gábor· Apr 1, 2020, 22:18 UTC · re: Đoàn Trần Công Danh · lore

Re: [PATCH v2 3/4] travis: build and test on Linux with musl libc and busybox

On Sun, Mar 29, 2020 at 05:12:31PM +0700, Đoàn Trần Công Danh wrote:
> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
Show 25 quoted lines
> diff --git a/ci/run-alpine-build.sh b/ci/run-alpine-build.sh
> new file mode 100755
> index 0000000000..c83df536e4
> --- /dev/null
> +++ b/ci/run-alpine-build.sh
> @@ -0,0 +1,31 @@
> +#!/bin/sh
> +#
> +# Build and test Git in Alpine Linux
> +#
> +# Usage:
> +#   run-alpine-build.sh <host-user-id>
> +#
> +
> +set -ex
> +
> +useradd () {
> +	adduser -D "$@"
> +}
> +
> +. "${0%/*}/lib-docker.sh"
> +
> +# Update packages to the latest available versions
> +apk add --update autoconf build-base curl-dev openssl-dev expat-dev \
> +	gettext pcre2-dev python3 musl-libintl >/dev/null

In 'ci/run-docker.sh' we run 'docker run' with a bunch of '--env ...' options to make some important environment variables available inside the container. At this point in this script all those variables are set to the expected values, but ...

> +# Build and test
> +su -m -l $CI_USER -c '

... but here, for some reason, those environment variables are not set anymore. This is bad, because this CI job then builds Git sequentially, runs the tests sequentially, runs the tests with 'make' instead of 'prove', and runs the tests without '-V -x'. IOW, it's slow, it produces a lot of useless output, it doesn't report all the failures, and doesn't tell us anything about the failures.

Show 6 quoted lines
> +	set -ex
> +	cd /usr/src/git
> +	test -n "$cache_dir" && ln -s "$cache_dir/.prove" t/.prove
> +	autoconf
> +	echo "PYTHON_PATH=/usr/bin/python3" >config.mak
> +	./configure --with-libpcre

The recommended way to build Git is without autoconf and configure. You can set the PYTHON_PATH and USE_LIBPCRE Makefile knobs in MAKEFLAGS in 'ci/lib.sh', but to be able to access MAKEFLAGS in the container you'll need to build this patch series on top of:

  https://public-inbox.org/git/20200401212151.15164-1-szeder.dev@gmail.com/
Show 21 quoted lines
> +	make
> +	make test
> +'
> diff --git a/ci/run-docker.sh b/ci/run-docker.sh
> index be698817cb..f203db03cf 100755
> --- a/ci/run-docker.sh
> +++ b/ci/run-docker.sh
> @@ -10,6 +10,10 @@ Linux32)
>  	CI_TARGET=linux32
>  	CI_CONTAINER="daald/ubuntu32:xenial"
>  	;;
> +linux-musl)
> +	CI_TARGET=alpine
> +	CI_CONTAINER=alpine
> +	;;
>  *)
>  	exit 1 ;;
>  esac
> -- 
> 2.26.0.302.g234993491e
>
Danh Doan· Apr 2, 2020, 01:42 UTC · re: SZEDER Gábor · lore

Re: [PATCH v2 3/4] travis: build and test on Linux with musl libc and busybox

On 2020-04-02 00:18:35+0200, SZEDER Gábor <szeder.dev@gmail.com> wrote:
Show 43 quoted lines
> On Sun, Mar 29, 2020 at 05:12:31PM +0700, Đoàn Trần Công Danh wrote:
> > Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> 
> > diff --git a/ci/run-alpine-build.sh b/ci/run-alpine-build.sh
> > new file mode 100755
> > index 0000000000..c83df536e4
> > --- /dev/null
> > +++ b/ci/run-alpine-build.sh
> > @@ -0,0 +1,31 @@
> > +#!/bin/sh
> > +#
> > +# Build and test Git in Alpine Linux
> > +#
> > +# Usage:
> > +#   run-alpine-build.sh <host-user-id>
> > +#
> > +
> > +set -ex
> > +
> > +useradd () {
> > +	adduser -D "$@"
> > +}
> > +
> > +. "${0%/*}/lib-docker.sh"
> > +
> > +# Update packages to the latest available versions
> > +apk add --update autoconf build-base curl-dev openssl-dev expat-dev \
> > +	gettext pcre2-dev python3 musl-libintl >/dev/null
> 
> In 'ci/run-docker.sh' we run 'docker run' with a bunch of '--env ...'
> options to make some important environment variables available
> inside the container.  At this point in this script all those
> variables are set to the expected values, but ...
> 
> > +# Build and test
> > +su -m -l $CI_USER -c '
> 
> ... but here, for some reason, those environment variables are not set
> anymore.  This is bad, because this CI job then builds Git
> sequentially, runs the tests sequentially, runs the tests with 'make'
> instead of 'prove', and runs the tests without '-V -x'.  IOW, it's
> slow, it produces a lot of useless output, it doesn't report all the
> failures, and doesn't tell us anything about the failures.
At this point, I tempted to change this to
	su -m -l $CI_USER -c /usr/src/git/ci/run-build-and-tests.sh
instead.

But, after digging into ci/lib.sh, I found too many setup for "$CI_*", let choose your path instead.

Show 35 quoted lines
> > +	set -ex
> > +	cd /usr/src/git
> > +	test -n "$cache_dir" && ln -s "$cache_dir/.prove" t/.prove
> > +	autoconf
> > +	echo "PYTHON_PATH=/usr/bin/python3" >config.mak
> > +	./configure --with-libpcre
> 
> The recommended way to build Git is without autoconf and configure.
> You can set the PYTHON_PATH and USE_LIBPCRE Makefile knobs in
> MAKEFLAGS in 'ci/lib.sh', but to be able to access MAKEFLAGS in the
> container you'll need to build this patch series on top of:
> 
>   https://public-inbox.org/git/20200401212151.15164-1-szeder.dev@gmail.com/
> 
> > +	make
> > +	make test
> > +'
> > diff --git a/ci/run-docker.sh b/ci/run-docker.sh
> > index be698817cb..f203db03cf 100755
> > --- a/ci/run-docker.sh
> > +++ b/ci/run-docker.sh
> > @@ -10,6 +10,10 @@ Linux32)
> >  	CI_TARGET=linux32
> >  	CI_CONTAINER="daald/ubuntu32:xenial"
> >  	;;
> > +linux-musl)
> > +	CI_TARGET=alpine
> > +	CI_CONTAINER=alpine
> > +	;;
> >  *)
> >  	exit 1 ;;
> >  esac
> > -- 
> > 2.26.0.302.g234993491e
> > 
-- 
Danh
Johannes Schindelin· Apr 7, 2020, 14:53 UTC · re: SZEDER Gábor · lore

Re: [PATCH v2 3/4] travis: build and test on Linux with musl libc and busybox

Hi Gábor,
On Thu, 2 Apr 2020, SZEDER Gábor wrote:
Show 11 quoted lines
> On Sun, Mar 29, 2020 at 05:12:31PM +0700, Đoàn Trần Công Danh wrote:
> > Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
>
> > +	set -ex
> > +	cd /usr/src/git
> > +	test -n "$cache_dir" && ln -s "$cache_dir/.prove" t/.prove
> > +	autoconf
> > +	echo "PYTHON_PATH=/usr/bin/python3" >config.mak
> > +	./configure --with-libpcre
>
> The recommended way to build Git is without autoconf and configure.
That is news to me.

My understanding still is that `make` is the recommended way to build Git, and `./configure` is only for those who want to use autoconf.

It seems that the `INSTALL` file agrees with my understanding:
-- snip --
                Git installation

Normally you can just do "make" followed by "make install", and that will install the git programs in your own ~/bin/ directory. If you want to do a global install, you can do

        $ make prefix=/usr all doc info ;# as yourself
        # make prefix=/usr install install-doc install-html install-info
        # ;# as root

(or prefix=/usr/local, of course). Just like any program suite that uses $prefix, the built results have some paths encoded, which are derived from $prefix, so "make all; make prefix=/usr install" would not work.

The beginning of the Makefile documents many variables that affect the way git is built. You can override them either from the command line, or in a config.mak file.

Alternatively you can use autoconf generated ./configure script to set up install paths (via config.mak.autogen), so you can write instead

        $ make configure ;# as yourself
        $ ./configure --prefix=/usr ;# as yourself
        $ make all doc ;# as yourself
        # make install install-doc install-html;# as root
-- snap --

If you think that I am wrong, I invite you to change the recommendation by proposing a patch to `INSTALL`, to change the current recommendation.

Ciao, Dscho

Junio C Hamano· Apr 7, 2020, 21:35 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 3/4] travis: build and test on Linux with musl libc and busybox

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
Show 8 quoted lines
>> The recommended way to build Git is without autoconf and configure.
>
> That is news to me.
>
> My understanding still is that `make` is the recommended way to build Git,
> and `./configure` is only for those who want to use autoconf.
>
> It seems that the `INSTALL` file agrees with my understanding:

Did you misread the sentence you quoted? It says "without", not "with", so I think you two are on the same page.

Johannes Schindelin· Apr 10, 2020, 13:38 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 3/4] travis: build and test on Linux with musl libc and busybox

Hi Junio,
On Tue, 7 Apr 2020, Junio C Hamano wrote:
Show 13 quoted lines
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
> >> The recommended way to build Git is without autoconf and configure.
> >
> > That is news to me.
> >
> > My understanding still is that `make` is the recommended way to build Git,
> > and `./configure` is only for those who want to use autoconf.
> >
> > It seems that the `INSTALL` file agrees with my understanding:
>
> Did you misread the sentence you quoted?  It says "without", not
> "with", so I think you two are on the same page.
Yes, I misread it. Thanks for pointing that out.

Ciao, Dscho

Đoàn Trần Công Danh· Mar 29, 2020, 10:12 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v2 4/4] t3402: use POSIX compliant regex(7)

`\?` is undefined for POSIX BRE, from: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09_03_02

Show 5 quoted lines
> The interpretation of an ordinary character preceded
> by an unescaped <backslash> ( '\\' ) is undefined, except for:
> - The characters ')', '(', '{', and '}'
> - The digits 1 to 9 inclusive
> - A character inside a bracket expression
This test is failing with busybox grep.
Fix it by using character class.
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
Needs to be applied on top of jt/rebase-allow-duplicate 
Cc: Jonathan Tan <jonathantanmy@google.com>
 t/t3402-rebase-merge.sh | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
Show changes to t/t3402-rebase-merge.sh +4 −4
diff --git a/t/t3402-rebase-merge.sh b/t/t3402-rebase-merge.sh
index 290c79e0f6..e8bab8102d 100755
--- a/t/t3402-rebase-merge.sh
+++ b/t/t3402-rebase-merge.sh
@@ -228,15 +228,15 @@ test_expect_success '--skip-cherry-pick-detection refrains from reading unneeded
 	git -C client rev-list --objects --all --missing=print >missing_list &&
 	MERGE_BASE_BLOB=$(git -C server rev-parse master^^:file.txt) &&
 	ADD_11_BLOB=$(git -C server rev-parse master^:file.txt) &&
-	grep "\\?$MERGE_BASE_BLOB" missing_list &&
-	grep "\\?$ADD_11_BLOB" missing_list &&
+	grep "[?]$MERGE_BASE_BLOB" missing_list &&
+	grep "[?]$ADD_11_BLOB" missing_list &&
 
 	git -C client rebase --merge --skip-cherry-pick-detection origin/master &&
 
 	# The blob from the merge base had to be fetched, but not "add 11"
 	git -C client rev-list --objects --all --missing=print >missing_list &&
-	! grep "\\?$MERGE_BASE_BLOB" missing_list &&
-	grep "\\?$ADD_11_BLOB" missing_list
+	! grep "[?]$MERGE_BASE_BLOB" missing_list &&
+	grep "[?]$ADD_11_BLOB" missing_list
 '
 
 test_done

-- 
2.26.0.302.g234993491e
Junio C Hamano· Mar 29, 2020, 16:23 UTC · re: Đoàn Trần Công Danh · lore

Re: [PATCH v2 0/4] Travis + Azure jobs for linux with musl libc

Đoàn Trần Công Danh  <congdanhqx@gmail.com> writes:
Show 9 quoted lines
> Change from v1:
> - fix spelling
> - run-docker.sh: use "jobname" environment variable instead of passing argument
> - add linux-musl job on Azure
> - Add 4th patch for jt/rebase-allow-duplicate (feel free to squash into
>  jt/rebase-allow-duplicate)
>
> The first 3 patches could be applied on top of master,
> but the last patch needs to be applied on top of jt/rebase-allow-duplicate
This note was very helpful.  Very much appreciated.

Please keep this a three-patch series ([1/4], [2/4] and [3/4] become [1/3], [2/3] and [3/3]), and make the fourth one a separate fix to the other topic. Even if we were not going to take this topic, the last one is an independently useful improvement.

I'll update jt/rebase-allow-duplicate with the last one, so no real harm done, but keeping the topics separate on the list would help reduce confusion.

Thanks.
Đoàn Trần Công Danh· Apr 2, 2020, 13:03 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v3 0/6] Travis + Azure jobs for linux with musl libc

Recently, we've un-broken git for Linux with musl libc, and we have a serie to fix false negative with busybox shell utils.

Add a CI job on Travis and Azure to make sure we won't break it again.

This is a nearly rewrite of this series, because there're GitHub Action allow running directly inside container.

=> I rewrite this series to prepare as much as possible for the GitHub Action series. => No range-diff

The first patch is coming from Szeder, Junio hasn't picked it up yet. And, this series depends on it.

Sample build without busybox fix series: https://travis-ci.org/github/sgn/git/builds/670097222

With busybox fix: https://travis-ci.org/github/sgn/git/builds/670103249

SZEDER Gábor (1):
  ci: make MAKEFLAGS available inside the Docker container in the
    Linux32 job
Đoàn Trần Công Danh (5):
  ci/lib-docker: preserve required environment variables
  ci/linux32: parameterise command to switch arch
  ci: refactor docker runner script
  ci/linux32: libify install-dependencies step
  travis: build and test on Linux with musl libc and busybox
 .travis.yml                                   | 10 ++++-
 azure-pipelines.yml                           | 39 ++++++++++++++++++-
 ci/install-docker-dependencies.sh             | 18 +++++++++
 ci/lib.sh                                     |  8 ++++
 ...n-linux32-build.sh => run-docker-build.sh} | 39 +++++++++++++------
 ci/{run-linux32-docker.sh => run-docker.sh}   | 28 ++++++++++---
 6 files changed, 121 insertions(+), 21 deletions(-)
 create mode 100755 ci/install-docker-dependencies.sh
 rename ci/{run-linux32-build.sh => run-docker-build.sh} (63%)
 rename ci/{run-linux32-docker.sh => run-docker.sh} (43%)
-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 2, 2020, 13:04 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v3 1/6] ci: make MAKEFLAGS available inside the Docker container in the Linux32 job

From: SZEDER Gábor <szeder.dev@gmail.com>

Once upon a time we ran 'make --jobs=2 ...' to build Git, its documentation, or to apply Coccinelle semantic patches. Then commit eaa62291ff (ci: inherit --jobs via MAKEFLAGS in run-build-and-tests, 2019-01-27) came along, and started using the MAKEFLAGS environment variable to centralize setting the number of parallel jobs in 'ci/libs.sh'. Alas, it forgot to update 'ci/run-linux32-docker.sh' to make MAKEFLAGS available inside the Docker container running the 32 bit Linux job, and, consequently, since then that job builds Git sequentially, and it ignores any Makefile knobs that we might set in MAKEFLAGS (though we don't set any for the 32 bit Linux job at the moment).

So update the 'docker run' invocation in 'ci/run-linux32-docker.sh' to make MAKEFLAGS available inside the Docker container as well. Set CC=gcc for the 32 bit Linux job, because that's the compiler installed in the 32 bit Linux Docker image that we use (Travis CI nowadays sets CC=clang by default, but clang is not installed in this image).

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 ci/lib.sh                | 3 +++
 ci/run-linux32-docker.sh | 1 +
 2 files changed, 4 insertions(+)
Show changes to 2 files +4 −0

ci/lib.sh, ci/run-linux32-docker.sh

diff --git a/ci/lib.sh b/ci/lib.sh
index c3a8cd2104..d637825222 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -198,6 +198,9 @@ osx-clang|osx-gcc)
 GIT_TEST_GETTEXT_POISON)
 	export GIT_TEST_GETTEXT_POISON=true
 	;;
+Linux32)
+	CC=gcc
+	;;
 esac
 
 MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
diff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh
index 751acfcf8a..ebb18fa747 100755
--- a/ci/run-linux32-docker.sh
+++ b/ci/run-linux32-docker.sh
@@ -20,6 +20,7 @@ docker run \
 	--env GIT_PROVE_OPTS \
 	--env GIT_TEST_OPTS \
 	--env GIT_TEST_CLONE_2GB \
+	--env MAKEFLAGS \
 	--env cache_dir="$container_cache_dir" \
 	--volume "${PWD}:/usr/src/git" \
 	--volume "$cache_dir:$container_cache_dir" \

-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 2, 2020, 13:04 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v3 2/6] ci/lib-docker: preserve required environment variables

We're using "su -m" to preserve environment variables in the shell run by "su". But, that options will be ignored while "-l" (aka "--login") is specified.

Since we don't have interest in all environment variables, pass only those necessary variables to the inner script.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 ci/run-linux32-build.sh | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
Show changes to ci/run-linux32-build.sh +10 −3
diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh
index e3a193adbc..7f985615c2 100755
--- a/ci/run-linux32-build.sh
+++ b/ci/run-linux32-build.sh
@@ -51,10 +51,17 @@ else
 fi
 
 # Build and test
-linux32 --32bit i386 su -m -l $CI_USER -c '
+linux32 --32bit i386 su -m -l $CI_USER -c "
 	set -ex
+	export DEVELOPER='$DEVELOPER'
+	export DEFAULT_TEST_TARGET='$DEFAULT_TEST_TARGET'
+	export GIT_PROVE_OPTS='$GIT_PROVE_OPTS'
+	export GIT_TEST_OPTS='$GIT_TEST_OPTS'
+	export GIT_TEST_CLONE_2GB='$GIT_TEST_CLONE_2GB'
+	export MAKEFLAGS='$MAKEFLAGS'
+	export cache_dir='$cache_dir'
 	cd /usr/src/git
-	test -n "$cache_dir" && ln -s "$cache_dir/.prove" t/.prove
+	test -n '$cache_dir' && ln -s '$cache_dir/.prove' t/.prove
 	make
 	make test
-'
+"

-- 
2.26.0.334.g6536db25bb
SZEDER Gábor· Apr 3, 2020, 08:22 UTC · re: Đoàn Trần Công Danh · lore

Re: [PATCH v3 2/6] ci/lib-docker: preserve required environment variables

On Thu, Apr 02, 2020 at 08:04:01PM +0700, Đoàn Trần Công Danh wrote:
> We're using "su -m" to preserve environment variables in the shell run
> by "su". But, that options will be ignored while "-l" (aka "--login") is
> specified.

This is not true. See any previous runs of the 32 bit Linux job, which worked as expected, because none of these environment variables were cleared.

Show 36 quoted lines
> Since we don't have interest in all environment variables,
> pass only those necessary variables to the inner script.
> 
> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> ---
>  ci/run-linux32-build.sh | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh
> index e3a193adbc..7f985615c2 100755
> --- a/ci/run-linux32-build.sh
> +++ b/ci/run-linux32-build.sh
> @@ -51,10 +51,17 @@ else
>  fi
>  
>  # Build and test
> -linux32 --32bit i386 su -m -l $CI_USER -c '
> +linux32 --32bit i386 su -m -l $CI_USER -c "
>  	set -ex
> +	export DEVELOPER='$DEVELOPER'
> +	export DEFAULT_TEST_TARGET='$DEFAULT_TEST_TARGET'
> +	export GIT_PROVE_OPTS='$GIT_PROVE_OPTS'
> +	export GIT_TEST_OPTS='$GIT_TEST_OPTS'
> +	export GIT_TEST_CLONE_2GB='$GIT_TEST_CLONE_2GB'
> +	export MAKEFLAGS='$MAKEFLAGS'
> +	export cache_dir='$cache_dir'
>  	cd /usr/src/git
> -	test -n "$cache_dir" && ln -s "$cache_dir/.prove" t/.prove
> +	test -n '$cache_dir' && ln -s '$cache_dir/.prove' t/.prove
>  	make
>  	make test
> -'
> +"
> -- 
> 2.26.0.334.g6536db25bb
>
Danh Doan· Apr 3, 2020, 10:09 UTC · re: SZEDER Gábor · lore

Re: [PATCH v3 2/6] ci/lib-docker: preserve required environment variables

On 2020-04-03 10:22:54+0200, SZEDER Gábor <szeder.dev@gmail.com> wrote:
Show 8 quoted lines
> On Thu, Apr 02, 2020 at 08:04:01PM +0700, Đoàn Trần Công Danh wrote:
> > We're using "su -m" to preserve environment variables in the shell run
> > by "su". But, that options will be ignored while "-l" (aka "--login") is
> > specified.
> 
> This is not true.  See any previous runs of the 32 bit Linux job,
> which worked as expected, because none of these environment variables
> were cleared.
Different su have different behavior when combine "-m" and "-l"

util-linux's su has this as far as 60541961f, (docs: improve grammar, wording and formatting of su man page, 2013-10-12)

       -m, -p, --preserve-environment
              Preserve the entire environment, i.e., it does not set HOME,
              SHELL, USER nor LOGNAME.  This option is ignored if the option
              --login is specified.
https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git/tree/login-utils/su.1#n120
Ubuntu (our Linux32 builder), ships su by shadow-utils:
	Note that the default behavior for the environment is the following:
		The $HOME, $SHELL, $USER, $LOGNAME, $PATH,
		and $IFS environment variables are reset.
		If --login is not used, the environment is copied,
		except for the variables above.
		If --login is used, the $TERM, $COLORTERM, $DISPLAY, and
		$XAUTHORITY environment variables are copied if they were set.

There're no mentions of other variables, I _think_ our Linux32 works by accident.

Alpine ships su from busybox, this su ignores "-m" if "-l" is set.
-- 
Danh
SZEDER Gábor· Apr 3, 2020, 19:55 UTC · re: Danh Doan · lore

Re: [PATCH v3 2/6] ci/lib-docker: preserve required environment variables

On Fri, Apr 03, 2020 at 05:09:31PM +0700, Danh Doan wrote:
Show 36 quoted lines
> On 2020-04-03 10:22:54+0200, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> > On Thu, Apr 02, 2020 at 08:04:01PM +0700, Đoàn Trần Công Danh wrote:
> > > We're using "su -m" to preserve environment variables in the shell run
> > > by "su". But, that options will be ignored while "-l" (aka "--login") is
> > > specified.
> > 
> > This is not true.  See any previous runs of the 32 bit Linux job,
> > which worked as expected, because none of these environment variables
> > were cleared.
> 
> Different su have different behavior when combine "-m" and "-l"
> 
> util-linux's su has this as far as 60541961f, (docs: improve grammar,
> wording and formatting of su man page, 2013-10-12)
> 
>        -m, -p, --preserve-environment
>               Preserve the entire environment, i.e., it does not set HOME,
>               SHELL, USER nor LOGNAME.  This option is ignored if the option
>               --login is specified.
> 
> https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git/tree/login-utils/su.1#n120
> 
> Ubuntu (our Linux32 builder), ships su by shadow-utils:
> 
> 	Note that the default behavior for the environment is the following:
> 		The $HOME, $SHELL, $USER, $LOGNAME, $PATH,
> 		and $IFS environment variables are reset.
> 
> 		If --login is not used, the environment is copied,
> 		except for the variables above.
> 
> 		If --login is used, the $TERM, $COLORTERM, $DISPLAY, and
> 		$XAUTHORITY environment variables are copied if they were set.
> 
> There're no mentions of other variables, I _think_ our Linux32 works
> by accident.

We do know which image we use, and we do know how its 'su' behaves. I think relying on that is fine, and it's not just "works by accident".

> Alpine ships su from busybox, this su ignores "-m" if "-l" is set.

Then the commit message should specifically mention these behavior differences between different 'su' variants.

Đoàn Trần Công Danh· Apr 2, 2020, 13:04 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v3 3/6] ci/linux32: parameterise command to switch arch

In a later patch, the remaining of this command will be re-used for the CI job for linux with musl libc.

Allow customisation of the emulator, now.
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 ci/run-linux32-build.sh  | 13 +++++++++++--
 ci/run-linux32-docker.sh |  2 ++
 2 files changed, 13 insertions(+), 2 deletions(-)
Show changes to 2 files +13 −2

ci/run-linux32-build.sh, ci/run-linux32-docker.sh

diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh
index 7f985615c2..44bb332f64 100755
--- a/ci/run-linux32-build.sh
+++ b/ci/run-linux32-build.sh
@@ -14,8 +14,17 @@ then
 	exit 1
 fi
 
+case "$jobname" in
+Linux32)
+	switch_cmd="linux32 --32bit i386"
+	;;
+*)
+	exit 1
+	;;
+esac
+
 # Update packages to the latest available versions
-linux32 --32bit i386 sh -c '
+command $switch_cmd sh -c '
     apt update >/dev/null &&
     apt install -y build-essential libcurl4-openssl-dev libssl-dev \
 	libexpat-dev gettext python >/dev/null
@@ -51,7 +60,7 @@ else
 fi
 
 # Build and test
-linux32 --32bit i386 su -m -l $CI_USER -c "
+command $switch_cmd su -m -l $CI_USER -c "
 	set -ex
 	export DEVELOPER='$DEVELOPER'
 	export DEFAULT_TEST_TARGET='$DEFAULT_TEST_TARGET'
diff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh
index ebb18fa747..54186b6aa7 100755
--- a/ci/run-linux32-docker.sh
+++ b/ci/run-linux32-docker.sh
@@ -9,6 +9,7 @@ docker pull daald/ubuntu32:xenial
 
 # Use the following command to debug the docker build locally:
 # $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/bash daald/ubuntu32:xenial
+# root@container:/# export jobname=<jobname>
 # root@container:/# /usr/src/git/ci/run-linux32-build.sh <host-user-id>
 
 container_cache_dir=/tmp/travis-cache
@@ -21,6 +22,7 @@ docker run \
 	--env GIT_TEST_OPTS \
 	--env GIT_TEST_CLONE_2GB \
 	--env MAKEFLAGS \
+	--env jobname \
 	--env cache_dir="$container_cache_dir" \
 	--volume "${PWD}:/usr/src/git" \
 	--volume "$cache_dir:$container_cache_dir" \

-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 2, 2020, 13:04 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v3 4/6] ci: refactor docker runner script

We will support alpine check in docker later in this series.

While we're at it, tell people to run as root in podman, if podman is used as drop-in replacement for docker, because podman will map host-user to container's root, therefore, mapping their permission.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 .travis.yml                                   |  2 +-
 azure-pipelines.yml                           |  4 ++--
 ...n-linux32-build.sh => run-docker-build.sh} |  6 ++---
 ci/{run-linux32-docker.sh => run-docker.sh}   | 22 ++++++++++++++-----
 4 files changed, 22 insertions(+), 12 deletions(-)
 rename ci/{run-linux32-build.sh => run-docker-build.sh} (93%)
 rename ci/{run-linux32-docker.sh => run-docker.sh} (53%)
Show changes to 6 files +22 −12

.travis.yml, azure-pipelines.yml, ci/run-linux32-build.sh, ci/run-docker-build.sh, ci/run-linux32-docker.sh, ci/run-docker.sh

diff --git a/.travis.yml b/.travis.yml
index fc5730b085..069aeeff3c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -32,7 +32,7 @@ matrix:
       services:
         - docker
       before_install:
-      script: ci/run-linux32-docker.sh
+      script: ci/run-docker.sh
     - env: jobname=StaticAnalysis
       os: linux
       compiler:
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 675c3a43c9..11413f66f8 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -489,14 +489,14 @@ jobs:
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
 
        res=0
-       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-linux32-docker.sh || res=1
+       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" jobname=Linux32 bash -lxc ci/run-docker.sh || res=1
 
        sudo chmod a+r t/out/TEST-*.xml
        test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
 
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
        exit $res
-    displayName: 'ci/run-linux32-docker.sh'
+    displayName: 'jobname=Linux32 ci/run-docker.sh'
     env:
       GITFILESHAREPWD: $(gitfileshare.pwd)
   - task: PublishTestResults@2
diff --git a/ci/run-linux32-build.sh b/ci/run-docker-build.sh
similarity index 93%
rename from ci/run-linux32-build.sh
rename to ci/run-docker-build.sh
index 44bb332f64..a05b48c559 100755
--- a/ci/run-linux32-build.sh
+++ b/ci/run-docker-build.sh
@@ -1,16 +1,16 @@
 #!/bin/sh
 #
-# Build and test Git in a 32-bit environment
+# Build and test Git inside container
 #
 # Usage:
-#   run-linux32-build.sh <host-user-id>
+#   run-docker-build.sh <host-user-id>
 #
 
 set -ex
 
 if test $# -ne 1 || test -z "$1"
 then
-	echo >&2 "usage: run-linux32-build.sh <host-user-id>"
+	echo >&2 "usage: run-docker-build.sh <host-user-id>"
 	exit 1
 fi
 
diff --git a/ci/run-linux32-docker.sh b/ci/run-docker.sh
similarity index 53%
rename from ci/run-linux32-docker.sh
rename to ci/run-docker.sh
index 54186b6aa7..3881f99b53 100755
--- a/ci/run-linux32-docker.sh
+++ b/ci/run-docker.sh
@@ -1,16 +1,26 @@
 #!/bin/sh
 #
-# Download and run Docker image to build and test 32-bit Git
+# Download and run Docker image to build and test Git
 #
 
 . ${0%/*}/lib.sh
 
-docker pull daald/ubuntu32:xenial
+case "$jobname" in
+Linux32)
+	CI_CONTAINER="daald/ubuntu32:xenial"
+	;;
+*)
+	exit 1
+	;;
+esac
+
+docker pull "$CI_CONTAINER"
 
 # Use the following command to debug the docker build locally:
-# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/bash daald/ubuntu32:xenial
+# <host-user-id> must be 0 if podman is used as drop-in replacement for docker
+# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/sh "$CI_CONTAINER"
 # root@container:/# export jobname=<jobname>
-# root@container:/# /usr/src/git/ci/run-linux32-build.sh <host-user-id>
+# root@container:/# /usr/src/git/ci/run-docker-build.sh <host-user-id>
 
 container_cache_dir=/tmp/travis-cache
 
@@ -26,8 +36,8 @@ docker run \
 	--env cache_dir="$container_cache_dir" \
 	--volume "${PWD}:/usr/src/git" \
 	--volume "$cache_dir:$container_cache_dir" \
-	daald/ubuntu32:xenial \
-	/usr/src/git/ci/run-linux32-build.sh $(id -u $USER)
+	"$CI_CONTAINER" \
+	/usr/src/git/ci/run-docker-build.sh $(id -u $USER)
 
 check_unignored_build_artifacts
 

-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 2, 2020, 13:04 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v3 5/6] ci/linux32: libify install-dependencies step

In a later patch, we will add new Travis Job for linux-musl. Most of other code in this file could be reuse for that job.

Move the code to install dependencies to a common script. Should we add new CI system that can run directly in container, we can reuse this script for installation step.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 ci/install-docker-dependencies.sh | 14 ++++++++++++++
 ci/run-docker-build.sh            |  7 +------
 2 files changed, 15 insertions(+), 6 deletions(-)
 create mode 100755 ci/install-docker-dependencies.sh
Show changes to 2 files +15 −6

ci/install-docker-dependencies.sh, ci/run-docker-build.sh

diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh
new file mode 100755
index 0000000000..a104c61d29
--- /dev/null
+++ b/ci/install-docker-dependencies.sh
@@ -0,0 +1,14 @@
+#!/bin/sh
+#
+# Install dependencies required to build and test Git inside container
+#
+
+case "$jobname" in
+Linux32)
+	linux32 --32bit i386 sh -c '
+		apt update >/dev/null &&
+		apt install -y build-essential libcurl4-openssl-dev \
+			libssl-dev libexpat-dev gettext python >/dev/null
+	'
+	;;
+esac
diff --git a/ci/run-docker-build.sh b/ci/run-docker-build.sh
index a05b48c559..4a153492ba 100755
--- a/ci/run-docker-build.sh
+++ b/ci/run-docker-build.sh
@@ -23,12 +23,7 @@ Linux32)
 	;;
 esac
 
-# Update packages to the latest available versions
-command $switch_cmd sh -c '
-    apt update >/dev/null &&
-    apt install -y build-essential libcurl4-openssl-dev libssl-dev \
-	libexpat-dev gettext python >/dev/null
-'
+"${0%/*}/install-docker-dependencies.sh"
 
 # If this script runs inside a docker container, then all commands are
 # usually executed as root. Consequently, the host user might not be

-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 2, 2020, 13:04 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v3 6/6] travis: build and test on Linux with musl libc and busybox

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 .travis.yml                       |  8 +++++++
 azure-pipelines.yml               | 35 +++++++++++++++++++++++++++++++
 ci/install-docker-dependencies.sh |  4 ++++
 ci/lib.sh                         |  5 +++++
 ci/run-docker-build.sh            |  4 ++++
 ci/run-docker.sh                  |  3 +++
 6 files changed, 59 insertions(+)
Show changes to 6 files +59 −0

.travis.yml, azure-pipelines.yml, ci/install-docker-dependencies.sh, ci/lib.sh, ci/run-docker-build.sh, ci/run-docker.sh

diff --git a/.travis.yml b/.travis.yml
index 069aeeff3c..0cfc3c3428 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -33,6 +33,14 @@ matrix:
         - docker
       before_install:
       script: ci/run-docker.sh
+    - env: jobname=linux-musl
+      os: linux
+      compiler:
+      addons:
+      services:
+        - docker
+      before_install:
+      script: ci/run-docker.sh
     - env: jobname=StaticAnalysis
       os: linux
       compiler:
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 11413f66f8..84ecad76ec 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -514,6 +514,41 @@ jobs:
       PathtoPublish: t/failed-test-artifacts
       ArtifactName: failed-test-artifacts
 
+- job: linux_musl
+  displayName: linux-musl
+  condition: succeeded()
+  pool:
+    vmImage: ubuntu-latest
+  steps:
+  - bash: |
+       test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
+
+       res=0
+       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" jobname=linux-musl bash -lxc ci/run-docker.sh || res=1
+
+       sudo chmod a+r t/out/TEST-*.xml
+       test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
+
+       test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
+       exit $res
+    displayName: 'jobname=linux-musl ci/run-docker.sh'
+    env:
+      GITFILESHAREPWD: $(gitfileshare.pwd)
+  - task: PublishTestResults@2
+    displayName: 'Publish Test Results **/TEST-*.xml'
+    inputs:
+      mergeTestResults: true
+      testRunTitle: 'musl'
+      platform: Linux
+      publishRunAttachments: false
+    condition: succeededOrFailed()
+  - task: PublishBuildArtifacts@1
+    displayName: 'Publish trash directories of failed tests'
+    condition: failed()
+    inputs:
+      PathtoPublish: t/failed-test-artifacts
+      ArtifactName: failed-test-artifacts
+
 - job: static_analysis
   displayName: StaticAnalysis
   condition: succeeded()
diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh
index a104c61d29..26a6689766 100755
--- a/ci/install-docker-dependencies.sh
+++ b/ci/install-docker-dependencies.sh
@@ -11,4 +11,8 @@ Linux32)
 			libssl-dev libexpat-dev gettext python >/dev/null
 	'
 	;;
+linux-musl)
+	apk add --update build-base curl-dev openssl-dev expat-dev gettext \
+		pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null
+	;;
 esac
diff --git a/ci/lib.sh b/ci/lib.sh
index d637825222..87cd29bab6 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -201,6 +201,11 @@ GIT_TEST_GETTEXT_POISON)
 Linux32)
 	CC=gcc
 	;;
+linux-musl)
+	CC=gcc
+	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
+	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
+	;;
 esac
 
 MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
diff --git a/ci/run-docker-build.sh b/ci/run-docker-build.sh
index 4a153492ba..8d47a5fda3 100755
--- a/ci/run-docker-build.sh
+++ b/ci/run-docker-build.sh
@@ -18,6 +18,10 @@ case "$jobname" in
 Linux32)
 	switch_cmd="linux32 --32bit i386"
 	;;
+linux-musl)
+	switch_cmd=
+	useradd () { adduser -D "$@"; }
+	;;
 *)
 	exit 1
 	;;
diff --git a/ci/run-docker.sh b/ci/run-docker.sh
index 3881f99b53..37fa372052 100755
--- a/ci/run-docker.sh
+++ b/ci/run-docker.sh
@@ -9,6 +9,9 @@ case "$jobname" in
 Linux32)
 	CI_CONTAINER="daald/ubuntu32:xenial"
 	;;
+linux-musl)
+	CI_CONTAINER=alpine
+	;;
 *)
 	exit 1
 	;;

-- 
2.26.0.334.g6536db25bb
Junio C Hamano· Apr 2, 2020, 17:53 UTC · re: Đoàn Trần Công Danh · lore

Re: [PATCH v3 0/6] Travis + Azure jobs for linux with musl libc

Đoàn Trần Công Danh  <congdanhqx@gmail.com> writes:
Show 16 quoted lines
> This is a nearly rewrite of this series, because there're GitHub Action
> allow running directly inside container.
>
> => I rewrite this series to prepare as much as possible for the GitHub
> Action series.
> ...
>  .travis.yml                                   | 10 ++++-
>  azure-pipelines.yml                           | 39 ++++++++++++++++++-
>  ci/install-docker-dependencies.sh             | 18 +++++++++
>  ci/lib.sh                                     |  8 ++++
>  ...n-linux32-build.sh => run-docker-build.sh} | 39 +++++++++++++------
>  ci/{run-linux32-docker.sh => run-docker.sh}   | 28 ++++++++++---
>  6 files changed, 121 insertions(+), 21 deletions(-)
>  create mode 100755 ci/install-docker-dependencies.sh
>  rename ci/{run-linux32-build.sh => run-docker-build.sh} (63%)
>  rename ci/{run-linux32-docker.sh => run-docker.sh} (43%)

Thanks. The above diffstat makes me wonder if it makes more sense to do the topic from Dscho first to migrate existing CI targets to GitHub Actions and then add musl job to the ci suite on top? That way, we won't have to worry about azure-pipelines.yml at all here.

Danh Doan· Apr 3, 2020, 00:23 UTC · re: Junio C Hamano · lore

Re: [PATCH v3 0/6] Travis + Azure jobs for linux with musl libc

On 2020-04-02 10:53:35-0700, Junio C Hamano <gitster@pobox.com> wrote:
Show 23 quoted lines
> Đoàn Trần Công Danh  <congdanhqx@gmail.com> writes:
> 
> > This is a nearly rewrite of this series, because there're GitHub Action
> > allow running directly inside container.
> >
> > => I rewrite this series to prepare as much as possible for the GitHub
> > Action series.
> > ...
> >  .travis.yml                                   | 10 ++++-
> >  azure-pipelines.yml                           | 39 ++++++++++++++++++-
> >  ci/install-docker-dependencies.sh             | 18 +++++++++
> >  ci/lib.sh                                     |  8 ++++
> >  ...n-linux32-build.sh => run-docker-build.sh} | 39 +++++++++++++------
> >  ci/{run-linux32-docker.sh => run-docker.sh}   | 28 ++++++++++---
> >  6 files changed, 121 insertions(+), 21 deletions(-)
> >  create mode 100755 ci/install-docker-dependencies.sh
> >  rename ci/{run-linux32-build.sh => run-docker-build.sh} (63%)
> >  rename ci/{run-linux32-docker.sh => run-docker.sh} (43%)
> 
> Thanks.  The above diffstat makes me wonder if it makes more sense
> to do the topic from Dscho first to migrate existing CI targets to
> GitHub Actions and then add musl job to the ci suite on top?  That
> way, we won't have to worry about azure-pipelines.yml at all here.
You can ignore the change to azure-pipelines.yml in 6/6
to reduce noise about Azure (it'll be deleted by next series anyway).
And declare that this series is working
for Travis only (as same intention of v1). New diffstat:
---------------8<------------------
 .travis.yml                                      | 10 +++++-
 azure-pipelines.yml                              |  4 +--
 ci/install-docker-dependencies.sh                | 18 +++++++++++
 ci/lib.sh                                        |  8 +++++
 ci/{run-linux32-build.sh => run-docker-build.sh} | 39 ++++++++++++++++--------
 ci/{run-linux32-docker.sh => run-docker.sh}      | 28 +++++++++++++----
 6 files changed, 86 insertions(+), 21 deletions(-)
---------------->8----------------
In _my_ opinion, I still prefer have this series first.
But, if we prefer to have GitHub Action first:
- We'll need 5/6 moved to that series
- In the rebased of this series, we'll change about 10 lines in GitHub Action yml.
If people think it's better that way, please tell me, I could re-order it.
-- 
Danh
Đoàn Trần Công Danh· Apr 4, 2020, 01:08 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v4 0/6] Travis jobs for linux with musl libc

Cc: Junio C Hamano <gitster@pobox.com>,
 SZEDER Gábor <szeder.dev@gmail.com>,
 Eric Sunshine <sunshine@sunshineco.com>,
 Johannes Schindelin <johannes.schindelin@gmx.de>

Recently, we've un-broken git for Linux with musl libc, and we have a serie to fix false negative with busybox shell utils.

Add a CI job on Travis and Azure to make sure we won't break it again.
Change from v3:
* reword 2/6: adding su behavior in util-linux and busybox
* 6/6: Drop change to azure-pipelines.yml, and declare this series will
  be used for Travis only. Since Azure will be replaced by GitHub Action
  in a later series.

Hi Junio, The series for GitHub Actions will need to be rebased on this series again. 6/6 in that seriess will have UD conflicts. Please "git rm azure-pipelines.yml" to fix conflicts.

SZEDER Gábor (1):
  ci: make MAKEFLAGS available inside the Docker container in the
    Linux32 job
Đoàn Trần Công Danh (5):
  ci/lib-docker: preserve required environment variables
  ci/linux32: parameterise command to switch arch
  ci: refactor docker runner script
  ci/linux32: libify install-dependencies step
  travis: build and test on Linux with musl libc and busybox
 .travis.yml                                   | 10 ++++-
 azure-pipelines.yml                           |  4 +-
 ci/install-docker-dependencies.sh             | 18 +++++++++
 ci/lib.sh                                     |  8 ++++
 ...n-linux32-build.sh => run-docker-build.sh} | 39 +++++++++++++------
 ci/{run-linux32-docker.sh => run-docker.sh}   | 28 ++++++++++---
 6 files changed, 86 insertions(+), 21 deletions(-)
 create mode 100755 ci/install-docker-dependencies.sh
 rename ci/{run-linux32-build.sh => run-docker-build.sh} (63%)
 rename ci/{run-linux32-docker.sh => run-docker.sh} (43%)
Range-diff against v3:
1:  2fdce60075 = 1:  2fdce60075 ci: make MAKEFLAGS available inside the Docker container in the Linux32 job
2:  b7b079f559 ! 2:  db574b3ff9 ci/lib-docker: preserve required environment variables
    @@ Commit message
     
         We're using "su -m" to preserve environment variables in the shell run
         by "su". But, that options will be ignored while "-l" (aka "--login") is
    -    specified.
    +    specified in util-linux and busybox's su.
    +
    +    In a later patch this script will be reused for checking Git for Linux
    +    with musl libc on Alpine Linux, Alpine Linux uses "su" from busybox.
     
         Since we don't have interest in all environment variables,
         pass only those necessary variables to the inner script.
3:  8c8cf3eb24 = 3:  a13715245f ci/linux32: parameterise command to switch arch
4:  22cc7960c3 = 4:  b5de868c1e ci: refactor docker runner script
5:  2e0d54f81e = 5:  c39451ffe5 ci/linux32: libify install-dependencies step
6:  b61ed50cf6 ! 6:  231affae83 travis: build and test on Linux with musl libc and busybox
    @@ .travis.yml: matrix:
            os: linux
            compiler:
     
    - ## azure-pipelines.yml ##
    -@@ azure-pipelines.yml: jobs:
    -       PathtoPublish: t/failed-test-artifacts
    -       ArtifactName: failed-test-artifacts
    - 
    -+- job: linux_musl
    -+  displayName: linux-musl
    -+  condition: succeeded()
    -+  pool:
    -+    vmImage: ubuntu-latest
    -+  steps:
    -+  - bash: |
    -+       test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
    -+
    -+       res=0
    -+       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" jobname=linux-musl bash -lxc ci/run-docker.sh || res=1
    -+
    -+       sudo chmod a+r t/out/TEST-*.xml
    -+       test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
    -+
    -+       test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
    -+       exit $res
    -+    displayName: 'jobname=linux-musl ci/run-docker.sh'
    -+    env:
    -+      GITFILESHAREPWD: $(gitfileshare.pwd)
    -+  - task: PublishTestResults@2
    -+    displayName: 'Publish Test Results **/TEST-*.xml'
    -+    inputs:
    -+      mergeTestResults: true
    -+      testRunTitle: 'musl'
    -+      platform: Linux
    -+      publishRunAttachments: false
    -+    condition: succeededOrFailed()
    -+  - task: PublishBuildArtifacts@1
    -+    displayName: 'Publish trash directories of failed tests'
    -+    condition: failed()
    -+    inputs:
    -+      PathtoPublish: t/failed-test-artifacts
    -+      ArtifactName: failed-test-artifacts
    -+
    - - job: static_analysis
    -   displayName: StaticAnalysis
    -   condition: succeeded()
    -
      ## ci/install-docker-dependencies.sh ##
     @@ ci/install-docker-dependencies.sh: Linux32)
      			libssl-dev libexpat-dev gettext python >/dev/null
-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 4, 2020, 01:08 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v4 1/6] ci: make MAKEFLAGS available inside the Docker container in the Linux32 job

From: SZEDER Gábor <szeder.dev@gmail.com>

Once upon a time we ran 'make --jobs=2 ...' to build Git, its documentation, or to apply Coccinelle semantic patches. Then commit eaa62291ff (ci: inherit --jobs via MAKEFLAGS in run-build-and-tests, 2019-01-27) came along, and started using the MAKEFLAGS environment variable to centralize setting the number of parallel jobs in 'ci/libs.sh'. Alas, it forgot to update 'ci/run-linux32-docker.sh' to make MAKEFLAGS available inside the Docker container running the 32 bit Linux job, and, consequently, since then that job builds Git sequentially, and it ignores any Makefile knobs that we might set in MAKEFLAGS (though we don't set any for the 32 bit Linux job at the moment).

So update the 'docker run' invocation in 'ci/run-linux32-docker.sh' to make MAKEFLAGS available inside the Docker container as well. Set CC=gcc for the 32 bit Linux job, because that's the compiler installed in the 32 bit Linux Docker image that we use (Travis CI nowadays sets CC=clang by default, but clang is not installed in this image).

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 ci/lib.sh                | 3 +++
 ci/run-linux32-docker.sh | 1 +
 2 files changed, 4 insertions(+)
Show changes to 2 files +4 −0

ci/lib.sh, ci/run-linux32-docker.sh

diff --git a/ci/lib.sh b/ci/lib.sh
index c3a8cd2104..d637825222 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -198,6 +198,9 @@ osx-clang|osx-gcc)
 GIT_TEST_GETTEXT_POISON)
 	export GIT_TEST_GETTEXT_POISON=true
 	;;
+Linux32)
+	CC=gcc
+	;;
 esac
 
 MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
diff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh
index 751acfcf8a..ebb18fa747 100755
--- a/ci/run-linux32-docker.sh
+++ b/ci/run-linux32-docker.sh
@@ -20,6 +20,7 @@ docker run \
 	--env GIT_PROVE_OPTS \
 	--env GIT_TEST_OPTS \
 	--env GIT_TEST_CLONE_2GB \
+	--env MAKEFLAGS \
 	--env cache_dir="$container_cache_dir" \
 	--volume "${PWD}:/usr/src/git" \
 	--volume "$cache_dir:$container_cache_dir" \

-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 4, 2020, 01:08 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v4 2/6] ci/lib-docker: preserve required environment variables

We're using "su -m" to preserve environment variables in the shell run by "su". But, that options will be ignored while "-l" (aka "--login") is specified in util-linux and busybox's su.

In a later patch this script will be reused for checking Git for Linux with musl libc on Alpine Linux, Alpine Linux uses "su" from busybox.

Since we don't have interest in all environment variables, pass only those necessary variables to the inner script.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 ci/run-linux32-build.sh | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
Show changes to ci/run-linux32-build.sh +10 −3
diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh
index e3a193adbc..7f985615c2 100755
--- a/ci/run-linux32-build.sh
+++ b/ci/run-linux32-build.sh
@@ -51,10 +51,17 @@ else
 fi
 
 # Build and test
-linux32 --32bit i386 su -m -l $CI_USER -c '
+linux32 --32bit i386 su -m -l $CI_USER -c "
 	set -ex
+	export DEVELOPER='$DEVELOPER'
+	export DEFAULT_TEST_TARGET='$DEFAULT_TEST_TARGET'
+	export GIT_PROVE_OPTS='$GIT_PROVE_OPTS'
+	export GIT_TEST_OPTS='$GIT_TEST_OPTS'
+	export GIT_TEST_CLONE_2GB='$GIT_TEST_CLONE_2GB'
+	export MAKEFLAGS='$MAKEFLAGS'
+	export cache_dir='$cache_dir'
 	cd /usr/src/git
-	test -n "$cache_dir" && ln -s "$cache_dir/.prove" t/.prove
+	test -n '$cache_dir' && ln -s '$cache_dir/.prove' t/.prove
 	make
 	make test
-'
+"

-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 4, 2020, 01:08 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v4 3/6] ci/linux32: parameterise command to switch arch

In a later patch, the remaining of this command will be re-used for the CI job for linux with musl libc.

Allow customisation of the emulator, now.
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 ci/run-linux32-build.sh  | 13 +++++++++++--
 ci/run-linux32-docker.sh |  2 ++
 2 files changed, 13 insertions(+), 2 deletions(-)
Show changes to 2 files +13 −2

ci/run-linux32-build.sh, ci/run-linux32-docker.sh

diff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh
index 7f985615c2..44bb332f64 100755
--- a/ci/run-linux32-build.sh
+++ b/ci/run-linux32-build.sh
@@ -14,8 +14,17 @@ then
 	exit 1
 fi
 
+case "$jobname" in
+Linux32)
+	switch_cmd="linux32 --32bit i386"
+	;;
+*)
+	exit 1
+	;;
+esac
+
 # Update packages to the latest available versions
-linux32 --32bit i386 sh -c '
+command $switch_cmd sh -c '
     apt update >/dev/null &&
     apt install -y build-essential libcurl4-openssl-dev libssl-dev \
 	libexpat-dev gettext python >/dev/null
@@ -51,7 +60,7 @@ else
 fi
 
 # Build and test
-linux32 --32bit i386 su -m -l $CI_USER -c "
+command $switch_cmd su -m -l $CI_USER -c "
 	set -ex
 	export DEVELOPER='$DEVELOPER'
 	export DEFAULT_TEST_TARGET='$DEFAULT_TEST_TARGET'
diff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh
index ebb18fa747..54186b6aa7 100755
--- a/ci/run-linux32-docker.sh
+++ b/ci/run-linux32-docker.sh
@@ -9,6 +9,7 @@ docker pull daald/ubuntu32:xenial
 
 # Use the following command to debug the docker build locally:
 # $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/bash daald/ubuntu32:xenial
+# root@container:/# export jobname=<jobname>
 # root@container:/# /usr/src/git/ci/run-linux32-build.sh <host-user-id>
 
 container_cache_dir=/tmp/travis-cache
@@ -21,6 +22,7 @@ docker run \
 	--env GIT_TEST_OPTS \
 	--env GIT_TEST_CLONE_2GB \
 	--env MAKEFLAGS \
+	--env jobname \
 	--env cache_dir="$container_cache_dir" \
 	--volume "${PWD}:/usr/src/git" \
 	--volume "$cache_dir:$container_cache_dir" \

-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 4, 2020, 01:08 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v4 4/6] ci: refactor docker runner script

We will support alpine check in docker later in this series.

While we're at it, tell people to run as root in podman, if podman is used as drop-in replacement for docker, because podman will map host-user to container's root, therefore, mapping their permission.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 .travis.yml                                   |  2 +-
 azure-pipelines.yml                           |  4 ++--
 ...n-linux32-build.sh => run-docker-build.sh} |  6 ++---
 ci/{run-linux32-docker.sh => run-docker.sh}   | 22 ++++++++++++++-----
 4 files changed, 22 insertions(+), 12 deletions(-)
 rename ci/{run-linux32-build.sh => run-docker-build.sh} (93%)
 rename ci/{run-linux32-docker.sh => run-docker.sh} (53%)
Show changes to 6 files +22 −12

.travis.yml, azure-pipelines.yml, ci/run-linux32-build.sh, ci/run-docker-build.sh, ci/run-linux32-docker.sh, ci/run-docker.sh

diff --git a/.travis.yml b/.travis.yml
index fc5730b085..069aeeff3c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -32,7 +32,7 @@ matrix:
       services:
         - docker
       before_install:
-      script: ci/run-linux32-docker.sh
+      script: ci/run-docker.sh
     - env: jobname=StaticAnalysis
       os: linux
       compiler:
diff --git a/azure-pipelines.yml b/azure-pipelines.yml
index 675c3a43c9..11413f66f8 100644
--- a/azure-pipelines.yml
+++ b/azure-pipelines.yml
@@ -489,14 +489,14 @@ jobs:
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1
 
        res=0
-       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" bash -lxc ci/run-linux32-docker.sh || res=1
+       sudo AGENT_OS="$AGENT_OS" BUILD_BUILDNUMBER="$BUILD_BUILDNUMBER" BUILD_REPOSITORY_URI="$BUILD_REPOSITORY_URI" BUILD_SOURCEBRANCH="$BUILD_SOURCEBRANCH" BUILD_SOURCEVERSION="$BUILD_SOURCEVERSION" SYSTEM_PHASENAME="$SYSTEM_PHASENAME" SYSTEM_TASKDEFINITIONSURI="$SYSTEM_TASKDEFINITIONSURI" SYSTEM_TEAMPROJECT="$SYSTEM_TEAMPROJECT" CC=$CC MAKEFLAGS="$MAKEFLAGS" jobname=Linux32 bash -lxc ci/run-docker.sh || res=1
 
        sudo chmod a+r t/out/TEST-*.xml
        test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts
 
        test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || sudo umount "$HOME/test-cache" || res=1
        exit $res
-    displayName: 'ci/run-linux32-docker.sh'
+    displayName: 'jobname=Linux32 ci/run-docker.sh'
     env:
       GITFILESHAREPWD: $(gitfileshare.pwd)
   - task: PublishTestResults@2
diff --git a/ci/run-linux32-build.sh b/ci/run-docker-build.sh
similarity index 93%
rename from ci/run-linux32-build.sh
rename to ci/run-docker-build.sh
index 44bb332f64..a05b48c559 100755
--- a/ci/run-linux32-build.sh
+++ b/ci/run-docker-build.sh
@@ -1,16 +1,16 @@
 #!/bin/sh
 #
-# Build and test Git in a 32-bit environment
+# Build and test Git inside container
 #
 # Usage:
-#   run-linux32-build.sh <host-user-id>
+#   run-docker-build.sh <host-user-id>
 #
 
 set -ex
 
 if test $# -ne 1 || test -z "$1"
 then
-	echo >&2 "usage: run-linux32-build.sh <host-user-id>"
+	echo >&2 "usage: run-docker-build.sh <host-user-id>"
 	exit 1
 fi
 
diff --git a/ci/run-linux32-docker.sh b/ci/run-docker.sh
similarity index 53%
rename from ci/run-linux32-docker.sh
rename to ci/run-docker.sh
index 54186b6aa7..3881f99b53 100755
--- a/ci/run-linux32-docker.sh
+++ b/ci/run-docker.sh
@@ -1,16 +1,26 @@
 #!/bin/sh
 #
-# Download and run Docker image to build and test 32-bit Git
+# Download and run Docker image to build and test Git
 #
 
 . ${0%/*}/lib.sh
 
-docker pull daald/ubuntu32:xenial
+case "$jobname" in
+Linux32)
+	CI_CONTAINER="daald/ubuntu32:xenial"
+	;;
+*)
+	exit 1
+	;;
+esac
+
+docker pull "$CI_CONTAINER"
 
 # Use the following command to debug the docker build locally:
-# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/bash daald/ubuntu32:xenial
+# <host-user-id> must be 0 if podman is used as drop-in replacement for docker
+# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/sh "$CI_CONTAINER"
 # root@container:/# export jobname=<jobname>
-# root@container:/# /usr/src/git/ci/run-linux32-build.sh <host-user-id>
+# root@container:/# /usr/src/git/ci/run-docker-build.sh <host-user-id>
 
 container_cache_dir=/tmp/travis-cache
 
@@ -26,8 +36,8 @@ docker run \
 	--env cache_dir="$container_cache_dir" \
 	--volume "${PWD}:/usr/src/git" \
 	--volume "$cache_dir:$container_cache_dir" \
-	daald/ubuntu32:xenial \
-	/usr/src/git/ci/run-linux32-build.sh $(id -u $USER)
+	"$CI_CONTAINER" \
+	/usr/src/git/ci/run-docker-build.sh $(id -u $USER)
 
 check_unignored_build_artifacts
 

-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 4, 2020, 01:08 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v4 5/6] ci/linux32: libify install-dependencies step

In a later patch, we will add new Travis Job for linux-musl. Most of other code in this file could be reuse for that job.

Move the code to install dependencies to a common script. Should we add new CI system that can run directly in container, we can reuse this script for installation step.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 ci/install-docker-dependencies.sh | 14 ++++++++++++++
 ci/run-docker-build.sh            |  7 +------
 2 files changed, 15 insertions(+), 6 deletions(-)
 create mode 100755 ci/install-docker-dependencies.sh
Show changes to 2 files +15 −6

ci/install-docker-dependencies.sh, ci/run-docker-build.sh

diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh
new file mode 100755
index 0000000000..a104c61d29
--- /dev/null
+++ b/ci/install-docker-dependencies.sh
@@ -0,0 +1,14 @@
+#!/bin/sh
+#
+# Install dependencies required to build and test Git inside container
+#
+
+case "$jobname" in
+Linux32)
+	linux32 --32bit i386 sh -c '
+		apt update >/dev/null &&
+		apt install -y build-essential libcurl4-openssl-dev \
+			libssl-dev libexpat-dev gettext python >/dev/null
+	'
+	;;
+esac
diff --git a/ci/run-docker-build.sh b/ci/run-docker-build.sh
index a05b48c559..4a153492ba 100755
--- a/ci/run-docker-build.sh
+++ b/ci/run-docker-build.sh
@@ -23,12 +23,7 @@ Linux32)
 	;;
 esac
 
-# Update packages to the latest available versions
-command $switch_cmd sh -c '
-    apt update >/dev/null &&
-    apt install -y build-essential libcurl4-openssl-dev libssl-dev \
-	libexpat-dev gettext python >/dev/null
-'
+"${0%/*}/install-docker-dependencies.sh"
 
 # If this script runs inside a docker container, then all commands are
 # usually executed as root. Consequently, the host user might not be

-- 
2.26.0.334.g6536db25bb
Đoàn Trần Công Danh· Apr 4, 2020, 01:08 UTC · re: Đoàn Trần Công Danh · lore

[PATCH v4 6/6] travis: build and test on Linux with musl libc and busybox

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 .travis.yml                       | 8 ++++++++
 ci/install-docker-dependencies.sh | 4 ++++
 ci/lib.sh                         | 5 +++++
 ci/run-docker-build.sh            | 4 ++++
 ci/run-docker.sh                  | 3 +++
 5 files changed, 24 insertions(+)
Show changes to 5 files +24 −0

.travis.yml, ci/install-docker-dependencies.sh, ci/lib.sh, ci/run-docker-build.sh, ci/run-docker.sh

diff --git a/.travis.yml b/.travis.yml
index 069aeeff3c..0cfc3c3428 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -33,6 +33,14 @@ matrix:
         - docker
       before_install:
       script: ci/run-docker.sh
+    - env: jobname=linux-musl
+      os: linux
+      compiler:
+      addons:
+      services:
+        - docker
+      before_install:
+      script: ci/run-docker.sh
     - env: jobname=StaticAnalysis
       os: linux
       compiler:
diff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh
index a104c61d29..26a6689766 100755
--- a/ci/install-docker-dependencies.sh
+++ b/ci/install-docker-dependencies.sh
@@ -11,4 +11,8 @@ Linux32)
 			libssl-dev libexpat-dev gettext python >/dev/null
 	'
 	;;
+linux-musl)
+	apk add --update build-base curl-dev openssl-dev expat-dev gettext \
+		pcre2-dev python3 musl-libintl perl-utils ncurses >/dev/null
+	;;
 esac
diff --git a/ci/lib.sh b/ci/lib.sh
index d637825222..87cd29bab6 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -201,6 +201,11 @@ GIT_TEST_GETTEXT_POISON)
 Linux32)
 	CC=gcc
 	;;
+linux-musl)
+	CC=gcc
+	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
+	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
+	;;
 esac
 
 MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
diff --git a/ci/run-docker-build.sh b/ci/run-docker-build.sh
index 4a153492ba..8d47a5fda3 100755
--- a/ci/run-docker-build.sh
+++ b/ci/run-docker-build.sh
@@ -18,6 +18,10 @@ case "$jobname" in
 Linux32)
 	switch_cmd="linux32 --32bit i386"
 	;;
+linux-musl)
+	switch_cmd=
+	useradd () { adduser -D "$@"; }
+	;;
 *)
 	exit 1
 	;;
diff --git a/ci/run-docker.sh b/ci/run-docker.sh
index 3881f99b53..37fa372052 100755
--- a/ci/run-docker.sh
+++ b/ci/run-docker.sh
@@ -9,6 +9,9 @@ case "$jobname" in
 Linux32)
 	CI_CONTAINER="daald/ubuntu32:xenial"
 	;;
+linux-musl)
+	CI_CONTAINER=alpine
+	;;
 *)
 	exit 1
 	;;

-- 
2.26.0.334.g6536db25bb
Junio C Hamano· Apr 5, 2020, 20:39 UTC · re: Đoàn Trần Công Danh · lore

Re: [PATCH v4 0/6] Travis jobs for linux with musl libc

Đoàn Trần Công Danh  <congdanhqx@gmail.com> writes:
> The series for GitHub Actions will need to be rebased on this series again.
> 6/6 in that seriess will have UD conflicts.
> Please "git rm azure-pipelines.yml" to fix conflicts.

If you want to keep doing this, please take over the ownership of both series and build one on top of the other. I asked you and Dscho to coordinate and work together, but Dscho seems to be comfortable with the idea of letting you touch his series, so doing so would still count as you two working together ;-). I do not have a strong opinion which parts should come first (it is something that can be decided between you two which way is cleaner).

Thanks.
Johannes Schindelin· Apr 7, 2020, 14:55 UTC · re: Junio C Hamano · lore

Re: [PATCH v4 0/6] Travis jobs for linux with musl libc

Hi Junio,
On Sun, 5 Apr 2020, Junio C Hamano wrote:
Show 13 quoted lines
> Đoàn Trần Công Danh  <congdanhqx@gmail.com> writes:
>
> > The series for GitHub Actions will need to be rebased on this series again.
> > 6/6 in that seriess will have UD conflicts.
> > Please "git rm azure-pipelines.yml" to fix conflicts.
>
> If you want to keep doing this, please take over the ownership of
> both series and build one on top of the other.  I asked you and
> Dscho to coordinate and work together, but Dscho seems to be
> comfortable with the idea of letting you touch his series, so doing
> so would still count as you two working together ;-).  I do not have
> a strong opinion which parts should come first (it is something that
> can be decided between you two which way is cleaner).

For the record, Danh and I _are_ working together, and yes, I am totally comfortable with him taking the lead on submitting the patch series' iterations.

Thanks, Dscho

Junio C Hamano· Apr 7, 2020, 19:25 UTC · re: Johannes Schindelin · lore

Re: [PATCH v4 0/6] Travis jobs for linux with musl libc

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
Show 21 quoted lines
> Hi Junio,
>
> On Sun, 5 Apr 2020, Junio C Hamano wrote:
>
>> Đoàn Trần Công Danh  <congdanhqx@gmail.com> writes:
>>
>> > The series for GitHub Actions will need to be rebased on this series again.
>> > 6/6 in that seriess will have UD conflicts.
>> > Please "git rm azure-pipelines.yml" to fix conflicts.
>>
>> If you want to keep doing this, please take over the ownership of
>> both series and build one on top of the other.  I asked you and
>> Dscho to coordinate and work together, but Dscho seems to be
>> comfortable with the idea of letting you touch his series, so doing
>> so would still count as you two working together ;-).  I do not have
>> a strong opinion which parts should come first (it is something that
>> can be decided between you two which way is cleaner).
>
> For the record, Danh and I _are_ working together, and yes, I am totally
> comfortable with him taking the lead on submitting the patch series'
> iterations.
Good.  Thanks, both.

← back to recent threads