{"thread":{"id":"59335","subject":"[PATCH 0/4] Add a CI for unsigned char system","startedAt":"2023-03-03T13:47:00Z","lastAt":"2023-03-09T11:02:54Z","messageCount":9,"participants":["Đoàn Trần Công Danh","Jeff King","Chris Torek"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"472953","messageId":"cover.1677850517.git.congdanhqx@gmail.com","threadId":"59335","inReplyTo":null,"subject":"[PATCH 0/4] Add a CI for unsigned char system","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2023-03-03T13:46:02Z","receivedAt":"2023-03-03T13:47:00Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"Recently, we have a brokeness on system with unsigned char because most of\npeople are working with x86_64 which has signed char.\n\nThis series tries to add a CI system for a widely used system with signed\nchar, which is arm64 via circle-ci.\n\nIn order to register for git to participate in Circle CI, you need to:\n- Create an account and login into Circle CI, (I tried login with GitHub OAuth);\n- Register a project with Circle CI, and choose which config.yml we would like\n  to be used for Circle CI\n\nCircle CI seems to allow 40000 (Forty thousand credits per month), in my\ntesting, with resource_class: 'arm.medium', (which costs 10 credits per minute),\nit take 12m15s to finish, so that will be about 3000 run per month. I believe\narm.large, which is also available to free accounts and cost 20 credits per\nminute, doesn't cost effective but I haven't checked out.\n\nSample run:\n- Without the fix for unsigned char:\n  https://app.circleci.com/pipelines/gh/sgn/git/12/workflows/9b39391d-0d03-4669-86e6-2796fd671c43/jobs/10\n- Normal run:\n  https://app.circleci.com/pipelines/gh/sgn/git/14/workflows/b8745f8e-8a57-4215-847e-2af66c1dd3c7/jobs/12\n\nI'm not really sure if we want the 'store_test_results', which will upload the\ntest report in the 'TESTS' tab:\nhttps://app.circleci.com/pipelines/gh/sgn/git/14/workflows/b8745f8e-8a57-4215-847e-2af66c1dd3c7/jobs/12/tests\nand provides a test insights like this:\nhttps://app.circleci.com/insights/gh/sgn/git/workflows/workflow/tests?branch=circle-ci\n\nĐoàn Trần Công Danh (4):\n  ci/lib.sh: remove an useless break\n  ci/install-dependencies: libify p4 and git-lfs installation\n  ci/install-dependencies: install git-lfs for arm64 build\n  ci: add arm64 CI environment via CircleCI\n\n .circleci/config.yml       | 17 +++++++++++++++\n ci/install-dependencies.sh | 42 ++++++++++++++++++++++++++++++--------\n ci/lib.sh                  | 30 ++++++++++++++++++++++-----\n 3 files changed, 75 insertions(+), 14 deletions(-)\n create mode 100644 .circleci/config.yml\n\n-- \n2.40.0.rc1.2.gd15644fe02\n\n"},{"id":"472954","messageId":"c7312724f8a7ec9ccf63fb0621a996ebad3c6128.1677850518.git.congdanhqx@gmail.com","threadId":"59335","inReplyTo":"cover.1677850517.git.congdanhqx@gmail.com","subject":"[PATCH 1/4] ci/lib.sh: remove an useless break","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2023-03-03T13:46:03Z","receivedAt":"2023-03-03T13:47:01Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"This statement was silently skipped for a long time.\n\n> ci/lib.sh: line 232: break: only meaningful in a `for', `while', or `until' loop\n\nSigned-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>\n---\n ci/lib.sh | 5 -----\n 1 file changed, 5 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex db7105e8a8..661076e70b 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -227,11 +227,6 @@ export SKIP_DASHED_BUILT_INS=YesPlease\n \n case \"$runs_on_pool\" in\n ubuntu-*)\n-\tif test \"$jobname\" = \"linux-gcc-default\"\n-\tthen\n-\t\tbreak\n-\tfi\n-\n \tPYTHON_PACKAGE=python2\n \tif test \"$jobname\" = linux-gcc\n \tthen\n-- \n2.40.0.rc1.2.gd15644fe02\n\n"},{"id":"472955","messageId":"13137839f39bdcbc56eae9e96894b5ebe4072374.1677850518.git.congdanhqx@gmail.com","threadId":"59335","inReplyTo":"cover.1677850517.git.congdanhqx@gmail.com","subject":"[PATCH 2/4] ci/install-dependencies: libify p4 and git-lfs installation","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2023-03-03T13:46:04Z","receivedAt":"2023-03-03T13:47:03Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"In a later change, we will install git-lfs for CircleCI test\nenvironment. Let move those installations out in order to reuse them.\n\nSigned-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>\n---\n ci/install-dependencies.sh | 38 +++++++++++++++++++++++++++++---------\n 1 file changed, 29 insertions(+), 9 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 4f407530d3..9a32af8968 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -6,16 +6,12 @@\n . ${0%/*}/lib.sh\n \n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n-LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n+LFSWHENCE=https://github.com/github/git-lfs/releases/download\n UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n  tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n  libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n-case \"$runs_on_pool\" in\n-ubuntu-*)\n-\tsudo apt-get -q update\n-\tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n-\t\t$UBUNTU_COMMON_PKGS $CC_PACKAGE $PYTHON_PACKAGE\n+_install_linux_p4() {\n \tmkdir --parents \"$P4_PATH\"\n \tpushd \"$P4_PATH\"\n \t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n@@ -23,12 +19,36 @@ ubuntu-*)\n \t\tchmod u+x p4d\n \t\tchmod u+x p4\n \tpopd\n+}\n+\n+_install_linux_git_lfs() {\n+\tlocal whence fname\n+\twhence=\"$1\"\n+\tfname=\"$2\"\n \tmkdir --parents \"$GIT_LFS_PATH\"\n \tpushd \"$GIT_LFS_PATH\"\n-\t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\t\ttar --extract --gunzip --file \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n+\t\twget --quiet \"$whence/$fname\"\n+\t\ttar --extract --gunzip --strip-components=1 --file \"$fname\"\n \tpopd\n+}\n+\n+case \"$runs_on_pool\" in\n+ubuntu-*)\n+\tsudo apt-get -q update\n+\tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n+\t\t$UBUNTU_COMMON_PKGS $CC_PACKAGE $PYTHON_PACKAGE\n+\n+\tcase \"$(uname -m)\" in\n+\tx86_64)\n+\t\t_install_linux_p4\n+\t\t_install_linux_git_lfs \"$LFSWHENCE/v${LINUX_GIT_LFS_VERSION}\" \\\n+\t\t\t\"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n+\t\t;;\n+\t*)\n+\t\techo \"Not supported architecture\" >&2\n+\t\texit 1\n+\t\t;;\n+\tesac\n \t;;\n macos-*)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n-- \n2.40.0.rc1.2.gd15644fe02\n\n"},{"id":"472956","messageId":"f8663249619f519157a19a2fa8f386dbbe23faba.1677850518.git.congdanhqx@gmail.com","threadId":"59335","inReplyTo":"cover.1677850517.git.congdanhqx@gmail.com","subject":"[PATCH 3/4] ci/install-dependencies: install git-lfs for arm64 build","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2023-03-03T13:46:05Z","receivedAt":"2023-03-03T13:47:10Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>\n---\n ci/install-dependencies.sh | 4 ++++\n ci/lib.sh                  | 6 ++++++\n 2 files changed, 10 insertions(+)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 9a32af8968..6d587ffc54 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -44,6 +44,10 @@ ubuntu-*)\n \t\t_install_linux_git_lfs \"$LFSWHENCE/v${LINUX_GIT_LFS_VERSION}\" \\\n \t\t\t\"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t\t;;\n+\taarch64)\n+\t\t_install_linux_git_lfs \"$LFSWHENCE/v${LINUX_GIT_LFS_VERSION}\" \\\n+\t\t\t\"git-lfs-linux-arm64-v$LINUX_GIT_LFS_VERSION.tar.gz\"\n+\t\t;;\n \t*)\n \t\techo \"Not supported architecture\" >&2\n \t\texit 1\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 661076e70b..3c19267a56 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -242,6 +242,12 @@ ubuntu-*)\n \t# image.\n \t# Keep that in mind when you encounter a broken OS X build!\n \texport LINUX_GIT_LFS_VERSION=\"1.5.2\"\n+\t# The Linux arm64 build installs yet another recent version, since\n+\t# above version doesn't ship prebuilt binaries for arm64.\n+\tif test \"$(uname -m)\" = aarch64\n+\tthen\n+\t\tLINUX_GIT_LFS_VERSION=\"3.3.0\"\n+\tfi\n \n \tP4_PATH=\"$HOME/custom/p4\"\n \tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n-- \n2.40.0.rc1.2.gd15644fe02\n\n"},{"id":"472957","messageId":"db5c58fb3ee114b56f110dcd20f938e77d8b476f.1677850518.git.congdanhqx@gmail.com","threadId":"59335","inReplyTo":"cover.1677850517.git.congdanhqx@gmail.com","subject":"[PATCH 4/4] ci: add arm64 CI environment via CircleCI","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2023-03-03T13:46:06Z","receivedAt":"2023-03-03T13:47:11Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>\n---\n .circleci/config.yml | 17 +++++++++++++++++\n ci/lib.sh            | 19 +++++++++++++++++++\n 2 files changed, 36 insertions(+)\n create mode 100644 .circleci/config.yml\n\ndiff --git a/.circleci/config.yml b/.circleci/config.yml\nnew file mode 100644\nindex 0000000000..2f5e0966b7\n--- /dev/null\n+++ b/.circleci/config.yml\n@@ -0,0 +1,17 @@\n+# .circleci/config.yml\n+jobs:\n+  build:\n+    machine:\n+      image: ubuntu-2004:2022.04.1\n+    resource_class: arm.medium\n+    environment:\n+      jobname: linux-gcc\n+      runs_on_pool: ubuntu-arm64\n+    steps:\n+    - checkout\n+    - run: ci/install-dependencies.sh\n+    - run:\n+        name: Build and Tests\n+        command: ci/run-build-and-tests.sh\n+    - store_test_results:\n+        path: t/out\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 3c19267a56..8b0c727950 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -201,6 +201,25 @@ then\n \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n \ttest windows != \"$CI_OS_NAME\" ||\n \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n+elif test \"$CIRCLECI\" = true\n+then\n+\tCI_TYPE=circleci\n+\tCI_BRANCH=\"$CIRCLE_BRANCH\"\n+\tCI_COMMIT=\"$CIRCLE_SHA1\"\n+\tCI_JOB_ID=\"$CIRCLE_WORKFLOW_JOB_ID\"\n+\tCI_JOB_NUMBER=\"$CIRCLE_BUILD_NUM\"\n+\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n+\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n+\tCI_REPO_SLUG=\"$(expr \"$CIRCLE_REPOSITORY_URL\" : '.*:\\([^/]*/[^/]*\\)\\.git$')\"\n+\tCC=\"${CC:-gcc}\"\n+\n+\t# use a subdirectory of the cache dir (because the file share is shared\n+\t# among *all* phases)\n+\tcache_dir=\"$HOME/none\"\n+\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n+\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n else\n \techo \"Could not identify CI type\" >&2\n \tenv >&2\n-- \n2.40.0.rc1.2.gd15644fe02\n\n"},{"id":"472958","messageId":"ZAH8McPd56NSAzOx@danh.dev","threadId":"59335","inReplyTo":"db5c58fb3ee114b56f110dcd20f938e77d8b476f.1677850518.git.congdanhqx@gmail.com","subject":"Re: [PATCH 4/4] ci: add arm64 CI environment via CircleCI","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2023-03-03T13:54:57Z","receivedAt":"2023-03-03T13:55:10Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2023-03-03 20:46:06+0700, Đoàn Trần Công Danh <congdanhqx@gmail.com> wrote:\n> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>\n> ---\n>  .circleci/config.yml | 17 +++++++++++++++++\n>  ci/lib.sh            | 19 +++++++++++++++++++\n>  2 files changed, 36 insertions(+)\n>  create mode 100644 .circleci/config.yml\n> \n> diff --git a/.circleci/config.yml b/.circleci/config.yml\n> new file mode 100644\n> index 0000000000..2f5e0966b7\n> --- /dev/null\n> +++ b/.circleci/config.yml\n> @@ -0,0 +1,17 @@\n> +# .circleci/config.yml\n> +jobs:\n> +  build:\n> +    machine:\n> +      image: ubuntu-2004:2022.04.1\n> +    resource_class: arm.medium\n> +    environment:\n> +      jobname: linux-gcc\n> +      runs_on_pool: ubuntu-arm64\n> +    steps:\n> +    - checkout\n> +    - run: ci/install-dependencies.sh\n> +    - run:\n> +        name: Build and Tests\n> +        command: ci/run-build-and-tests.sh\n> +    - store_test_results:\n> +        path: t/out\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 3c19267a56..8b0c727950 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -201,6 +201,25 @@ then\n>  \tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n>  \ttest windows != \"$CI_OS_NAME\" ||\n>  \tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n> +elif test \"$CIRCLECI\" = true\n> +then\n> +\tCI_TYPE=circleci\n> +\tCI_BRANCH=\"$CIRCLE_BRANCH\"\n> +\tCI_COMMIT=\"$CIRCLE_SHA1\"\n> +\tCI_JOB_ID=\"$CIRCLE_WORKFLOW_JOB_ID\"\n> +\tCI_JOB_NUMBER=\"$CIRCLE_BUILD_NUM\"\n> +\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n\nOops, this is wrong, it should be:\n\n\tCI_OS_NAME=\"$(uname -s)-$(uname -m)\"\n\n> +\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n> +\tCI_REPO_SLUG=\"$(expr \"$CIRCLE_REPOSITORY_URL\" : '.*:\\([^/]*/[^/]*\\)\\.git$')\"\n> +\tCC=\"${CC:-gcc}\"\n> +\n> +\t# use a subdirectory of the cache dir (because the file share is shared\n> +\t# among *all* phases)\n> +\tcache_dir=\"$HOME/none\"\n> +\n> +\texport GIT_PROVE_OPTS=\"--timer --jobs 10\"\n> +\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n> +\tMAKEFLAGS=\"$MAKEFLAGS --jobs=10\"\n>  else\n>  \techo \"Could not identify CI type\" >&2\n>  \tenv >&2\n> -- \n> 2.40.0.rc1.2.gd15644fe02\n> \n\n-- \nDanh\n"},{"id":"473269","messageId":"ZAmsikbYncUbNjD2@coredump.intra.peff.net","threadId":"59335","inReplyTo":"cover.1677850517.git.congdanhqx@gmail.com","subject":"Re: [PATCH 0/4] Add a CI for unsigned char system","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-03-09T09:53:14Z","receivedAt":"2023-03-09T09:53:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Mar 03, 2023 at 08:46:02PM +0700, Đoàn Trần Công Danh wrote:\n\n> Recently, we have a brokeness on system with unsigned char because most of\n> people are working with x86_64 which has signed char.\n> \n> This series tries to add a CI system for a widely used system with signed\n> char, which is arm64 via circle-ci.\n\nI like the overall goal, but I'm not wild about having another CI\nprovider. That requires people logging in there, and then dealing with\npossible credit overages, etc.\n\nI wonder what the timeline is for GitHub Actions getting arm64 support.\nIt looks like there are images for linux/arm64, but no runners yet,\naccording to:\n\n  https://github.com/actions/runner-images/issues/5631\n\nYou can point it at your own runners, and some people in that thread\nmentioned a third-party service which provides arm machines. That\ndoesn't get out of the \"oops, credits\" handling, but it would at least\nkeep the CI results all together.\n\nI dunno. Another option I saw suggested is using qemu within a regular\nGitHub runner. I have no idea if that would be painfully slow or what,\nbut it looks like people have even written actions to handle it:\n\n  https://github.com/docker/setup-qemu-action\n\n-Peff\n"},{"id":"473271","messageId":"CAPx1GveS2=8-rh6PVFsOXLvJ4iZGzeo9uUdG=wcbophxiDam0g@mail.gmail.com","threadId":"59335","inReplyTo":"ZAmsikbYncUbNjD2@coredump.intra.peff.net","subject":"Re: [PATCH 0/4] Add a CI for unsigned char system","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2023-03-09T10:26:34Z","receivedAt":"2023-03-09T10:27:32Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Thu, Mar 9, 2023 at 1:54 AM Jeff King <peff@peff.net> wrote:\n> I like the overall goal, but I'm not wild about having another CI\n> provider.\n\nSo, why not do an x86 build with `-funsigned-char`? Seems to work with\nboth gcc and clang.\n\nChris\n"},{"id":"473277","messageId":"ZAm7mpf36IOCqgN4@coredump.intra.peff.net","threadId":"59335","inReplyTo":"CAPx1GveS2=8-rh6PVFsOXLvJ4iZGzeo9uUdG=wcbophxiDam0g@mail.gmail.com","subject":"Re: [PATCH 0/4] Add a CI for unsigned char system","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-03-09T10:57:30Z","receivedAt":"2023-03-09T11:02:54Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Mar 09, 2023 at 02:26:34AM -0800, Chris Torek wrote:\n\n> On Thu, Mar 9, 2023 at 1:54 AM Jeff King <peff@peff.net> wrote:\n> > I like the overall goal, but I'm not wild about having another CI\n> > provider.\n> \n> So, why not do an x86 build with `-funsigned-char`? Seems to work with\n> both gcc and clang.\n\nYeah, that would be even simpler. Though IMHO \"unsigned char\" is only\none interesting difference to be checking. Another would be having a\nplatform where unaligned access isn't tolerated. It would be nice to\nhave a big-endian platform, too, but I'm not sure if arm is a good fit\nthere (my impression is that it can be run in either mode?).\n\nOn the other hand, I think Ævar does periodically run on the gcc build\nmachines, which includes examples of each (including aarch64). And this\nparticular bug was found pretty quickly (within a week of it hitting\nnext, and only a day after hitting master). So while it might be nice to\nhave more immediate CI feedback, it does seem like the old \"if the\nplatform matters, somebody will try it and report the problem\" strategy\nstill works, too.\n\n-Peff\n"}]}