{"thread":{"id":"61266","subject":"[PATCH 00/12] t: exercise Git/JGit reftable compatibility","startedAt":"2024-04-04T13:25:07Z","lastAt":"2024-05-06T05:53:42Z","messageCount":106,"participants":["Patrick Steinhardt","Han-Wen Nienhuys","Eric Sunshine","Junio C Hamano","Jeff King","Josh Steadmon","Toon claes","Justin Tobler"],"isPatch":true,"patchVersion":1,"patchTotal":12},"messages":[{"id":"492234","messageId":"cover.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":null,"subject":"[PATCH 00/12] t: exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:00Z","receivedAt":"2024-04-04T13:25:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nwhile the reftable backend is a recent addition to Git, it has been part\nof JGit since 2017 already. Given that there are essentially two\ndifferent implementations of the reftable format now there is a very\nreal risk of these two diverge and become incompatible with each other,\nwhich would be a shame.\n\nThis patch series addresses this risk by introducing compatibility tests\nwhich assert that both Git and JGit can access reftables written by the\nrespective other implementation.\n\nThe patch series is structured as follows:\n\n - Patches 1-8 merge \"install-docker-dependencies.sh\" into\n \"install-dependencies.sh\". This is done so that both CI job flavors\n have the same dependencies and thus the same test coverage available\n without always having to maintain them both.\n\n - Patch 9 makes JGit available.\n\n - Patch 10 starts running backend-specific tests in all jobs, and\n patch 11 addresses a portability issue surfaced by this.\n\n - Patch 12 adds a very basic compatibility test suite for Git/JGit\n reftables. I mostly consider this as a proof of concept, it should\n likely be extended over time.\n\nThese compatibility tests surface three findings:\n\n - JGit does not support reftables format v2, which was added to\n support the SHA256 object format.\n\n - JGit cannot read reflogs written by itself when starting from an\n unborn branch. This smells like a bug in JGit to me where it\n misinterprets reflog entries with a zero object ID as new OID, but I\n didn't dig any deeper yet.\n\n - JGit is incompatible with split indices because it cannot handle\n 'link' DIRC entries. This is unrelated to reftables though.\n\nI have tested the CI changes against both GitLab [1] and GitHub [2]. The\nmacOS test failures on GitHub are caused by the recent curl regression.\n\n[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/123\n[2]: https://github.com/git/git/pull/1696\n\nPatrick\n\nPatrick Steinhardt (12):\n ci: rename \"runs_on_pool\" to \"distro\"\n ci: expose distro name in dockerized GitHub jobs\n ci: allow skipping sudo on dockerized jobs\n ci: drop duplicate package installation for \"linux-gcc-default\"\n ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"\n ci: merge custom PATH directories\n ci: merge scripts which install dependencies\n ci: make Perforce binaries executable for all users\n ci: install JGit dependency\n t06xx: always execute backend-specific tests\n t0610: fix non-portable variable assignment\n t0612: add tests to exercise Git/JGit reftable compatibility\n\n .github/workflows/main.yml | 8 +-\n .gitlab-ci.yml | 4 +-\n ci/install-dependencies.sh | 100 ++++++++++++++-------\n ci/install-docker-dependencies.sh | 46 ----------\n ci/lib.sh | 14 ++-\n t/t0600-reffiles-backend.sh | 8 +-\n t/t0601-reffiles-pack-refs.sh | 9 +-\n t/t0610-reftable-basics.sh | 12 ++-\n t/t0612-reftable-jgit-compatibility.sh | 115 +++++++++++++++++++++++++\n 9 files changed, 208 insertions(+), 108 deletions(-)\n delete mode 100755 ci/install-docker-dependencies.sh\n create mode 100755 t/t0612-reftable-jgit-compatibility.sh\n\n\nbase-commit: 7774cfed6261ce2900c84e55906da708c711d601\n-- \n2.44.GIT\n\n"},{"id":"492235","messageId":"e6181295490ea02aae9269dca533d83f037d6960.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 01/12] ci: rename \"runs_on_pool\" to \"distro\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:04Z","receivedAt":"2024-04-04T13:25:08Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"runs_on_pool\" environment variable is used by our CI scripts to\ndistinguish the different kinds of operating systems. It is quite\nspecific to GitHub Actions though and not really a descriptive name.\n\nRename the variable to \"distro\" to clarify its intent.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 2 +-\n ci/install-dependencies.sh | 2 +-\n ci/lib.sh | 4 ++--\n 3 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 3428773b09..684ef5c00d 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -303,7 +303,7 @@ jobs:\n CC: ${{matrix.vector.cc}}\n CC_PACKAGE: ${{matrix.vector.cc_package}}\n jobname: ${{matrix.vector.jobname}}\n- runs_on_pool: ${{matrix.vector.pool}}\n+ distro: ${{matrix.vector.pool}}\n runs-on: ${{matrix.vector.pool}}\n steps:\n - uses: actions/checkout@v4\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex b4e22de3cb..7d247b5ef4 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -11,7 +11,7 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n-case \"$runs_on_pool\" in\n+case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 0a73fc7bd1..d882250db5 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -279,7 +279,7 @@ then\n \n \tcache_dir=\"$HOME/none\"\n \n-\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n+\tdistro=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n \tJOBS=$(nproc)\n else\n \techo \"Could not identify CI type\" >&2\n@@ -318,7 +318,7 @@ export DEFAULT_TEST_TARGET=prove\n export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n-case \"$runs_on_pool\" in\n+case \"$distro\" in\n ubuntu-*)\n \tif test \"$jobname\" = \"linux-gcc-default\"\n \tthen\n-- \n2.44.GIT\n\n"},{"id":"492236","messageId":"e3e2b7cd503bc769dbe5667c73b51d4612071440.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 02/12] ci: expose distro name in dockerized GitHub jobs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:09Z","receivedAt":"2024-04-04T13:25:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Expose a distro name in dockerized jobs. This will be used in a\nsubsequent commit where we merge the installation scripts for dockerized\nand non-dockerized jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 684ef5c00d..71cd4e5486 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -342,12 +342,16 @@ jobs:\n vector:\n - jobname: linux-musl\n image: alpine\n+ distro: alpine-latest\n - jobname: linux32\n image: daald/ubuntu32:xenial\n+ distro: ubuntu32-16.04\n - jobname: pedantic\n image: fedora\n+ distro: fedora-latest\n env:\n jobname: ${{matrix.vector.jobname}}\n+ distro: ${{matrix.vector.distro}}\n runs-on: ubuntu-latest\n container: ${{matrix.vector.image}}\n steps:\n-- \n2.44.GIT\n\n"},{"id":"492237","messageId":"8abc9ad6a70a09aae0a350499ee1f3a79796fe8d.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 03/12] ci: allow skipping sudo on dockerized jobs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:13Z","receivedAt":"2024-04-04T13:25:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our \"install-dependencies.sh\" script is executed by non-dockerized jobs\nto install dependencies. These jobs don't run with \"root\" permissions,\nbut with a separate user. Consequently, we need to use sudo(8) there to\nelevate permissions when installing packages.\n\nWe're about to merge \"install-docker-dependencies.sh\" into that script\nthough, and our Docker containers do run as \"root\". Using sudo(8) is\nthus unnecessary there, even though it would be harmless. On some images\nlike Alpine Linux though there is no sudo(8) available by default, which\nwould consequently break the build.\n\nAdapt the script to make \"sudo\" a no-op when running as \"root\" user.\nThis allows us to easily reuse the script for our dockerized jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7d247b5ef4..7dfd3e50ed 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -11,6 +11,17 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n+# Make sudo a no-op and execute the command directly when running as root.\n+# While using sudo would be fine on most platforms when we are root already,\n+# some platforms like e.g. Alpine Linux do not have sudo available by default\n+# and would thus break.\n+if test \"$(id -u)\" -eq 0\n+then\n+\tsudo () {\n+\t\t\"$@\"\n+\t}\n+fi\n+\n case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n-- \n2.44.GIT\n\n"},{"id":"492238","messageId":"7cf253862540bdcbcf6928a9ad64dadddbc943ca.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 04/12] ci: drop duplicate package installation for \"linux-gcc-default\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:17Z","receivedAt":"2024-04-04T13:25:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"linux-gcc-default\" job installs common Ubuntu packages. This is\nalready done in the distro-specific switch, so we basically duplicate\nthe effort here.\n\nDrop the duplicate package installations and inline the variable that\ncontains those common packages.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 17 ++++++++---------\n 1 file changed, 8 insertions(+), 9 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7dfd3e50ed..fad53aac96 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -7,9 +7,6 @@\n \n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n-UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n- tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n- libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n # Make sudo a no-op and execute the command directly when running as root.\n # While using sudo would be fine on most platforms when we are root already,\n@@ -25,8 +22,13 @@ fi\n case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n-\tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n-\t\t$UBUNTU_COMMON_PKGS $CC_PACKAGE $PYTHON_PACKAGE\n+\tsudo apt-get -q -y install \\\n+\t\tlanguage-pack-is libsvn-perl apache2 \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev \\\n+\t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n+\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n+\t\t$CC_PACKAGE $PYTHON_PACKAGE\n+\n \tmkdir --parents \"$P4_PATH\"\n \tpushd \"$P4_PATH\"\n \t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n@@ -34,6 +36,7 @@ ubuntu-*)\n \t\tchmod u+x p4d\n \t\tchmod u+x p4\n \tpopd\n+\n \tmkdir --parents \"$GIT_LFS_PATH\"\n \tpushd \"$GIT_LFS_PATH\"\n \t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n@@ -83,10 +86,6 @@ Documentation)\n \ttest -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n \tsudo gem install --version 1.5.8 asciidoctor\n \t;;\n-linux-gcc-default)\n-\tsudo apt-get -q update\n-\tsudo apt-get -q -y install $UBUNTU_COMMON_PKGS\n-\t;;\n esac\n \n if type p4d >/dev/null 2>&1 && type p4 >/dev/null 2>&1\n-- \n2.44.GIT\n\n"},{"id":"492239","messageId":"38e64224e2f0aa4d777308050ed8d513ae162607.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 05/12] ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:22Z","receivedAt":"2024-04-04T13:25:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to merge the \"install-docker-dependencies.sh\" script into\n\"install-dependencies.sh\". This will also move our Alpine-based jobs\nover to use the latter script. This script uses the Bash shell though,\nwhich is not available by default on Alpine Linux.\n\nRefactor \"install-dependencies.sh\" to use \"/bin/sh\" instead of Bash.\nThis requires us to get rid of the pushd/popd invocations, which are\nreplaced by some more elaborate commands that download or extract\nexecutables right to where they are needed.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 28 +++++++++++-----------------\n 1 file changed, 11 insertions(+), 17 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex fad53aac96..7bcccc96fd 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -1,4 +1,4 @@\n-#!/usr/bin/env bash\n+#!/bin/sh\n #\n # Install dependencies required to build and test Git on Linux and macOS\n #\n@@ -30,19 +30,14 @@ ubuntu-*)\n \t\t$CC_PACKAGE $PYTHON_PACKAGE\n \n \tmkdir --parents \"$P4_PATH\"\n-\tpushd \"$P4_PATH\"\n-\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n-\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\t\tchmod u+x p4d\n-\t\tchmod u+x p4\n-\tpopd\n+\twget --quiet --directory-prefix=\"$P4_PATH\" \\\n+\t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n+\tchmod u+x \"$P4_PATH/p4d\" \"$P4_PATH/p4\"\n \n \tmkdir --parents \"$GIT_LFS_PATH\"\n-\tpushd \"$GIT_LFS_PATH\"\n-\t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\t\ttar --extract --gunzip --file \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n-\tpopd\n+\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n+\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n+\trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n macos-*)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n@@ -53,11 +48,10 @@ macos-*)\n \tbrew link --force gettext\n \n \tmkdir -p \"$P4_PATH\"\n-\tpushd \"$P4_PATH\"\n-\t\twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n-\t\ttar -xf helix-core-server.tgz &&\n-\t\tsudo xattr -d com.apple.quarantine p4 p4d 2>/dev/null || true\n-\tpopd\n+\twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n+\ttar -xf helix-core-server.tgz -C \"$P4_PATH\" p4 p4d &&\n+\tsudo xattr -d com.apple.quarantine \"$P4_PATH/p4\" \"$P4_PATH/p4d\" 2>/dev/null || true\n+\trm helix-core-server.tgz\n \n \tif test -n \"$CC_PACKAGE\"\n \tthen\n-- \n2.44.GIT\n\n"},{"id":"492240","messageId":"196dab460a3be1e028924a88dfa794547c082bb1.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 06/12] ci: merge custom PATH directories","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:27Z","receivedAt":"2024-04-04T13:25:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're downloading various executables required by our tests. Each of\nthese executables goes into its own directory, which is then appended to\nthe PATH variable. Consequently, whenever we add a new dependency and\nthus a new directory, we would have to adapt to this change in several\nplaces.\n\nRefactor this to instead put all binaries into a single directory.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 15 +++++++--------\n ci/lib.sh | 10 +++-------\n 2 files changed, 10 insertions(+), 15 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7bcccc96fd..46b9efb2d5 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -29,14 +29,13 @@ ubuntu-*)\n \t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n \t\t$CC_PACKAGE $PYTHON_PACKAGE\n \n-\tmkdir --parents \"$P4_PATH\"\n-\twget --quiet --directory-prefix=\"$P4_PATH\" \\\n+\tmkdir --parents \"$CUSTOM_PATH\"\n+\twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n \t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\tchmod u+x \"$P4_PATH/p4d\" \"$P4_PATH/p4\"\n+\tchmod u+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n \n-\tmkdir --parents \"$GIT_LFS_PATH\"\n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n+\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n macos-*)\n@@ -47,10 +46,10 @@ macos-*)\n \tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \n-\tmkdir -p \"$P4_PATH\"\n+\tmkdir -p \"$CUSTOM_PATH\"\n \twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n-\ttar -xf helix-core-server.tgz -C \"$P4_PATH\" p4 p4d &&\n-\tsudo xattr -d com.apple.quarantine \"$P4_PATH/p4\" \"$P4_PATH/p4d\" 2>/dev/null || true\n+\ttar -xf helix-core-server.tgz -C \"$CUSTOM_PATH\" p4 p4d &&\n+\tsudo xattr -d com.apple.quarantine \"$CUSTOM_PATH/p4\" \"$CUSTOM_PATH/p4d\" 2>/dev/null || true\n \trm helix-core-server.tgz\n \n \tif test -n \"$CC_PACKAGE\"\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex d882250db5..4cce854bad 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -340,10 +340,6 @@ ubuntu-*)\n \t# image.\n \t# Keep that in mind when you encounter a broken OS X build!\n \texport LINUX_GIT_LFS_VERSION=\"1.5.2\"\n-\n-\tP4_PATH=\"$HOME/custom/p4\"\n-\tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n-\texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n macos-*)\n \tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n@@ -351,12 +347,12 @@ macos-*)\n \tthen\n \t\tMAKEFLAGS=\"$MAKEFLAGS APPLE_COMMON_CRYPTO_SHA1=Yes\"\n \tfi\n-\n-\tP4_PATH=\"$HOME/custom/p4\"\n-\texport PATH=\"$P4_PATH:$PATH\"\n \t;;\n esac\n \n+CUSTOM_PATH=\"$HOME/path\"\n+export PATH=\"$CUSTOM_PATH:$PATH\"\n+\n case \"$jobname\" in\n linux32)\n \tCC=gcc\n-- \n2.44.GIT\n\n"},{"id":"492241","messageId":"668553e18f59dea04b734f931e5b48d5d35c247d.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 07/12] ci: merge scripts which install dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:31Z","receivedAt":"2024-04-04T13:25:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different scripts which install dependencies, one for\ndockerized jobs and one for non-dockerized ones. Naturally, these\nscripts have quite some duplication. Furthermore, either of these\nscripts is missing some test dependencies that the respective other\nscript has, thus reducing test coverage.\n\nMerge those two scripts such that there is a single source of truth for\ntest dependencies, only.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 2 +-\n .gitlab-ci.yml | 4 +--\n ci/install-dependencies.sh | 32 ++++++++++++++++++---\n ci/install-docker-dependencies.sh | 46 -------------------------------\n 4 files changed, 31 insertions(+), 53 deletions(-)\n delete mode 100755 ci/install-docker-dependencies.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 71cd4e5486..5838986895 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -359,7 +359,7 @@ jobs:\n if: matrix.vector.jobname != 'linux32'\n - uses: actions/checkout@v1 # cannot be upgraded because Node.js Actions aren't supported in this container\n if: matrix.vector.jobname == 'linux32'\n- - run: ci/install-docker-dependencies.sh\n+ - run: ci/install-dependencies.sh\n - run: ci/run-build-and-tests.sh\n - name: print test failures\n if: failure() && env.FAILED_TEST_ARTIFACTS != ''\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex c0fa2fe90b..8ce0e1b4bd 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -10,7 +10,7 @@ workflow:\n test:linux:\n image: $image\n before_script:\n- - ./ci/install-docker-dependencies.sh\n+ - ./ci/install-dependencies.sh\n script:\n - useradd builder --create-home\n - chown -R builder \"${CI_PROJECT_DIR}\"\n@@ -98,7 +98,7 @@ static-analysis:\n variables:\n jobname: StaticAnalysis\n before_script:\n- - ./ci/install-docker-dependencies.sh\n+ - ./ci/install-dependencies.sh\n script:\n - ./ci/run-static-analysis.sh\n - ./ci/check-directional-formatting.bash\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 46b9efb2d5..f4eb125fd2 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -5,6 +5,8 @@\n \n . ${0%/*}/lib.sh\n \n+begin_group \"Install dependencies\"\n+\n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n \n@@ -20,14 +22,27 @@ then\n fi\n \n case \"$distro\" in\n+alpine-*)\n+\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n+\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n+\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n+\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n+\t;;\n+fedora-*)\n+\tdnf -yq update >/dev/null &&\n+\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\t;;\n ubuntu-*)\n+\t# Required so that apt doesn't wait for user input on certain packages.\n+\texport DEBIAN_FRONTEND=noninteractive\n+\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install \\\n-\t\tlanguage-pack-is libsvn-perl apache2 \\\n-\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev \\\n+\t\tlanguage-pack-is libsvn-perl apache2 cvs cvsps git gnupg subversion \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo \\\n \t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n-\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n-\t\t$CC_PACKAGE $PYTHON_PACKAGE\n+\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \\\n+\t\t${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE\n \n \tmkdir --parents \"$CUSTOM_PATH\"\n \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n@@ -38,6 +53,13 @@ ubuntu-*)\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n+ubuntu32-*)\n+\tsudo linux32 --32bit i386 sh -c '\n+\t\tapt update >/dev/null &&\n+\t\tapt install -y build-essential libcurl4-openssl-dev \\\n+\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n+\t'\n+\t;;\n macos-*)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n \t# Uncomment this if you want to run perf tests:\n@@ -97,3 +119,5 @@ then\n else\n \techo >&2 \"WARNING: git-lfs wasn't installed, see above for clues why\"\n fi\n+\n+end_group \"Install dependencies\"\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\ndeleted file mode 100755\nindex eb2c9e1eca..0000000000\n--- a/ci/install-docker-dependencies.sh\n+++ /dev/null\n@@ -1,46 +0,0 @@\n-#!/bin/sh\n-#\n-# Install dependencies required to build and test Git inside container\n-#\n-\n-. ${0%/*}/lib.sh\n-\n-begin_group \"Install dependencies\"\n-\n-case \"$jobname\" in\n-linux32)\n-\tlinux32 --32bit i386 sh -c '\n-\t\tapt update >/dev/null &&\n-\t\tapt install -y build-essential libcurl4-openssl-dev \\\n-\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n-\t'\n-\t;;\n-linux-musl)\n-\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n-\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n-\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n-\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n-\t;;\n-linux-*|StaticAnalysis)\n-\t# Required so that apt doesn't wait for user input on certain packages.\n-\texport DEBIAN_FRONTEND=noninteractive\n-\n-\tapt update -q &&\n-\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n-\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n-\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n-\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n-\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n-\n-\tif test \"$jobname\" = StaticAnalysis\n-\tthen\n-\t\tapt install -q -y coccinelle\n-\tfi\n-\t;;\n-pedantic)\n-\tdnf -yq update >/dev/null &&\n-\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n-\t;;\n-esac\n-\n-end_group \"Install dependencies\"\n-- \n2.44.GIT\n\n"},{"id":"492242","messageId":"22f86f8ccb9f3fb7f98ff57ddd09724fc9e44628.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 08/12] ci: make Perforce binaries executable for all users","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:35Z","receivedAt":"2024-04-04T13:25:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The Perforce binaries are only made executable for the current user. On\nGitLab CI though we execute tests as a different user than \"root\", and\nthus these binaries may not be executable by that test user.\n\nFix the setup so that we set the executable bits for all users.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex f4eb125fd2..068c478025 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -47,7 +47,7 @@ ubuntu-*)\n \tmkdir --parents \"$CUSTOM_PATH\"\n \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n \t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\tchmod u+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n+\tchmod a+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n \n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n-- \n2.44.GIT\n\n"},{"id":"492243","messageId":"1deded615eb7a26c3a4a47707dca6942a8a1732b.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 09/12] ci: install JGit dependency","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:39Z","receivedAt":"2024-04-04T13:25:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have some tests in t5310 that use JGit to verify that bitmaps can be\nread both by Git and by JGit. We do not execute these tests in our CI\njobs though because we don't make JGit available there. Consequently,\nthe tests basically bitrot because almost nobody is ever going to have\nJGit in their path.\n\nInstall JGit to plug this test gap.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 15 ++++++++++++++-\n 1 file changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 068c478025..3fefe134ea 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -9,6 +9,7 @@ begin_group \"Install dependencies\"\n \n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n+JGITWHENCE=https://repo.eclipse.org/content/groups/releases//org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh\n \n # Make sudo a no-op and execute the command directly when running as root.\n # While using sudo would be fine on most platforms when we are root already,\n@@ -39,7 +40,7 @@ ubuntu-*)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install \\\n \t\tlanguage-pack-is libsvn-perl apache2 cvs cvsps git gnupg subversion \\\n-\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo default-jre \\\n \t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n \t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \\\n \t\t${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE\n@@ -52,6 +53,9 @@ ubuntu-*)\n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n+\n+\twget --quiet \"$JGITWHENCE\" --output-document=\"$CUSTOM_PATH/jgit\"\n+\tchmod a+x \"$CUSTOM_PATH/jgit\"\n \t;;\n ubuntu32-*)\n \tsudo linux32 --32bit i386 sh -c '\n@@ -112,6 +116,7 @@ then\n else\n \techo >&2 \"WARNING: perforce wasn't installed, see above for clues why\"\n fi\n+\n if type git-lfs >/dev/null 2>&1\n then\n \techo \"$(tput setaf 6)Git-LFS Version$(tput sgr0)\"\n@@ -120,4 +125,12 @@ else\n \techo >&2 \"WARNING: git-lfs wasn't installed, see above for clues why\"\n fi\n \n+if type jgit >/dev/null 2>&1\n+then\n+\techo \"$(tput setaf 6)JGit Version$(tput sgr0)\"\n+\tjgit version\n+else\n+\techo >&2 \"WARNING: JGit wasn't installed, see above for clues why\"\n+fi\n+\n end_group \"Install dependencies\"\n-- \n2.44.GIT\n\n"},{"id":"492244","messageId":"51c45c879f6c683ab1bbf813ac26e1885e800d93.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 10/12] t06xx: always execute backend-specific tests","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:43Z","receivedAt":"2024-04-04T13:25:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The tests in t06xx exercise specific ref formats. Next to probing some\nbasic functionality, these tests also exercise other low-level details\nspecific to the format. Those tests are only executed though in case\n`GIT_TEST_DEFAULT_REF_FORMAT` is set to the ref format of the respective\nbackend-under-test.\n\nIdeally, we would run the full test matrix for ref formats such that our\ncomplete test suite is executed with every supported format on every\nsupported platform. This is quite an expensive undertaking though, and\nthus we only execute e.g. the \"reftable\" tests on macOS and Linux. As a\nresult, we basically have no test coverage for the \"reftable\" format at\nall on other platforms like Windows.\n\nAdapt these tests so that they override `GIT_TEST_DEFAULT_REF_FORMAT`,\nwhich means that they'll always execute. This increases test coverage on\nplatforms that don't run the full test matrix, which at least gives us\nsome basic test coverage on those platforms for the \"reftable\" format.\n\nThis of course comes at the cost of running those tests multiple times\non platforms where we do run the full test matrix. But arguably, this is\na good thing because it will also cause us to e.g. run those tests with\nthe address sanitizer and other non-standard parameters.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0600-reffiles-backend.sh | 8 ++------\n t/t0601-reffiles-pack-refs.sh | 9 +++------\n t/t0610-reftable-basics.sh | 9 +++------\n 3 files changed, 8 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t0600-reffiles-backend.sh b/t/t0600-reffiles-backend.sh\nindex 64214340e7..a390cffc80 100755\n--- a/t/t0600-reffiles-backend.sh\n+++ b/t/t0600-reffiles-backend.sh\n@@ -4,16 +4,12 @@ test_description='Test reffiles backend'\n \n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n-if ! test_have_prereq REFFILES\n-then\n-\tskip_all='skipping reffiles specific tests'\n-\ttest_done\n-fi\n-\n test_expect_success 'setup' '\n \tgit commit --allow-empty -m Initial &&\n \tC=$(git rev-parse HEAD) &&\ndiff --git a/t/t0601-reffiles-pack-refs.sh b/t/t0601-reffiles-pack-refs.sh\nindex c309d2bae8..157f79fe52 100755\n--- a/t/t0601-reffiles-pack-refs.sh\n+++ b/t/t0601-reffiles-pack-refs.sh\n@@ -9,18 +9,15 @@ test_description='git pack-refs should not change the branch semantic\n This test runs git pack-refs and git show-ref and checks that the branch\n semantic is still the same.\n '\n+\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n-if ! test_have_prereq REFFILES\n-then\n-\tskip_all='skipping reffiles specific tests'\n-\ttest_done\n-fi\n-\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 686781192e..aa9282007c 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -4,17 +4,14 @@\n #\n \n test_description='reftable basics'\n+\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n . ./test-lib.sh\n \n-if ! test_have_prereq REFTABLE\n-then\n-\tskip_all='skipping reftable tests; set GIT_TEST_DEFAULT_REF_FORMAT=reftable'\n-\ttest_done\n-fi\n-\n INVALID_OID=$(test_oid 001)\n \n test_expect_success 'init: creates basic reftable structures' '\n-- \n2.44.GIT\n\n"},{"id":"492245","messageId":"c2c2747ff57f68ccad8b509af037e1fc4a524fa1.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 11/12] t0610: fix non-portable variable assignment","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:48Z","receivedAt":"2024-04-04T13:25:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In `test_expect_perms()` we assign the output of a command to a variable\ndeclared via `local`. To assert that the command is actually successful\nwe also chain it with `&&`. This construct is seemingly not portable and\nmay fail with \"local: 1: bad variable name\".\n\nSplit up the variable declaration and assignment to fix this.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0610-reftable-basics.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex aa9282007c..3b1aa99e7c 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -80,8 +80,9 @@ test_expect_success 'init: reinitializing reftable with files backend fails' '\n test_expect_perms () {\n \tlocal perms=\"$1\"\n \tlocal file=\"$2\"\n-\tlocal actual=$(ls -l \"$file\") &&\n+\tlocal actual\n \n+\tactual=$(ls -l \"$file\") &&\n \tcase \"$actual\" in\n \t$perms*)\n \t\t: happy\n-- \n2.44.GIT\n\n"},{"id":"492246","messageId":"db66dd4155d80b714719e80ff90f64c1d36b97d0.1712235356.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-04T13:25:52Z","receivedAt":"2024-04-04T13:25:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While the reftable format is a recent introduction in Git, JGit already\nknows to read and write reftables since 2017. Given the complexity of\nthe format there is a very real risk of incompatibilities between those\ntwo implementations, which is something that we really want to avoid.\n\nAdd some basic tests that verify that reftables written by Git and JGit\ncan be read by the respective other implementation. For now this test\nsuite is rather small, only covering basic functionality. But it serves\nas a good starting point and can be extended over time.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0612-reftable-jgit-compatibility.sh | 115 +++++++++++++++++++++++++\n 1 file changed, 115 insertions(+)\n create mode 100755 t/t0612-reftable-jgit-compatibility.sh\n\ndiff --git a/t/t0612-reftable-jgit-compatibility.sh b/t/t0612-reftable-jgit-compatibility.sh\nnew file mode 100755\nindex 0000000000..f25ea880a0\n--- /dev/null\n+++ b/t/t0612-reftable-jgit-compatibility.sh\n@@ -0,0 +1,115 @@\n+#!/bin/sh\n+\n+test_description='reftables are compatible with JGit'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+# JGit does not support the 'link' DIRC extension.\n+GIT_TEST_SPLIT_INDEX=0\n+export GIT_TEST_SPLIT_INDEX\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq JGIT\n+then\n+\tskip_all='skipping reftable JGit tests'\n+\ttest_done\n+fi\n+\n+if ! test_have_prereq SHA1\n+then\n+\tskip_all='skipping reftable JGit tests; JGit does not support SHA256 reftables'\n+\ttest_done\n+fi\n+\n+test_commit_jgit () {\n+\ttouch \"$1\" &&\n+\tjgit add \"$1\" &&\n+\tjgit commit -m \"$1\"\n+}\n+\n+test_same_refs () {\n+\tgit show-ref --head >cgit.actual &&\n+\tjgit show-ref >jgit-tabs.actual &&\n+\ttr \"\\t\" \" \" jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_same_reflog () {\n+\tgit reflog \"$*\" >cgit.actual &&\n+\tjgit reflog \"$*\" >jgit-newline.actual &&\n+\tsed '/^$/d' jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_expect_success 'CGit repository can be read by JGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\ttest_same_refs &&\n+\t\ttest_same_reflog HEAD\n+\t)\n+'\n+\n+test_expect_success 'JGit repository can be read by CGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\t# JGit does not provide a way to create a reftable-enabled repository.\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttouch file &&\n+\t\tjgit add file &&\n+\t\tjgit commit -m \"initial commit\" &&\n+\n+\t\ttest_same_refs &&\n+\t\t# Interestingly, JGit cannot read its own reflog here. CGit can\n+\t\t# though.\n+\t\tprintf \"%s HEAD@{0}: commit (initial): initial commit\" \"$(git rev-parse --short HEAD)\" >expect &&\n+\t\tgit reflog HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'mixed writes from JGit and CGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit_jgit B &&\n+\t\ttest_commit C &&\n+\t\ttest_commit_jgit D &&\n+\n+\t\ttest_same_refs &&\n+\t\ttest_same_reflog HEAD\n+\t)\n+'\n+\n+test_expect_success 'JGit can read multi-level index' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\tawk \"\n+\t\t BEGIN {\n+\t\t\tprint \\\"start\\\";\n+\t\t\tfor (i = 0; i < 10000; i++)\n+\t\t\t printf \\\"create refs/heads/branch-%d HEAD\\n\\\", i;\n+\t\t\tprint \\\"commit\\\";\n+\t\t }\n+\t\t\" >input &&\n+\t\tgit update-ref --stdin Hi,\n> \n> while the reftable backend is a recent addition to Git, it has been part\n> of JGit since 2017 already. Given that there are essentially two\n> different implementations of the reftable format now there is a very\n> real risk of these two diverge and become incompatible with each other,\n> which would be a shame.\n> \n> This patch series addresses this risk by introducing compatibility tests\n> which assert that both Git and JGit can access reftables written by the\n> respective other implementation.\n> \n> The patch series is structured as follows:\n> \n> - Patches 1-8 merge \"install-docker-dependencies.sh\" into\n> \"install-dependencies.sh\". This is done so that both CI job flavors\n> have the same dependencies and thus the same test coverage available\n> without always having to maintain them both.\n> \n> - Patch 9 makes JGit available.\n> \n> - Patch 10 starts running backend-specific tests in all jobs, and\n> patch 11 addresses a portability issue surfaced by this.\n> \n> - Patch 12 adds a very basic compatibility test suite for Git/JGit\n> reftables. I mostly consider this as a proof of concept, it should\n> likely be extended over time.\n> \n> These compatibility tests surface three findings:\n> \n> - JGit does not support reftables format v2, which was added to\n> support the SHA256 object format.\n> \n> - JGit cannot read reflogs written by itself when starting from an\n> unborn branch. This smells like a bug in JGit to me where it\n> misinterprets reflog entries with a zero object ID as new OID, but I\n> didn't dig any deeper yet.\n> \n> - JGit is incompatible with split indices because it cannot handle\n> 'link' DIRC entries. This is unrelated to reftables though.\n> \n> I have tested the CI changes against both GitLab [1] and GitHub [2]. The\n> macOS test failures on GitHub are caused by the recent curl regression.\n> \n> [1]: https://gitlab.com/gitlab-org/git/-/merge_requests/123\n> [2]: https://github.com/git/git/pull/1696\n> \n> Patrick\n\nHrmpf, Cc'ing the jgit-dev@ list didn't work out as all my mails got\nrefused there due to myself not being signed up. Oh, well..\n\nPatrick\n"},{"id":"492273","messageId":"CAOw_e7b5wDKOcY9hSty+c+z-NWVUgri3h3ScF0qd1Y6cmfsyVA@mail.gmail.com","threadId":"61266","inReplyTo":"db66dd4155d80b714719e80ff90f64c1d36b97d0.1712235356.git.ps@pks.im","subject":"Re: [PATCH 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2024-04-04T21:40:29Z","receivedAt":"2024-04-04T21:40:41Z","isPatch":true,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"On Thu, Apr 4, 2024 at 5:01 PM Patrick Steinhardt wrote:\n> +\n> +test_same_refs () {\n> + git show-ref --head >cgit.actual &&\n> + jgit show-ref >jgit-tabs.actual &&\n\nThis seeks to the start and then iterates to the end, likely skipping\nindexes. If you want to test for indexes etc. you should also sample\nrandom refs from the namespace and see if they are returned correctly.\n\n> +test_expect_success 'JGit repository can be read by CGit' '\n> + test_when_finished \"rm -rf repo\" &&\n> + # JGit does not provide a way to create a reftable-enabled repository.\n\nYou can do \"jgit init && jgit convert-refstorage --format=reftable\"\n\n-- \nHan-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen\n"},{"id":"492284","messageId":"CAPig+cSb_cKXYaNCBpe9Uy_iGB_K2NXaw4d1hg5bPuVaCEWvjA@mail.gmail.com","threadId":"61266","inReplyTo":"c2c2747ff57f68ccad8b509af037e1fc4a524fa1.1712235356.git.ps@pks.im","subject":"Re: [PATCH 11/12] t0610: fix non-portable variable assignment","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-04-05T09:14:34Z","receivedAt":"2024-04-05T09:14:47Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, Apr 5, 2024 at 3:51 AM Patrick Steinhardt wrote:\n> In `test_expect_perms()` we assign the output of a command to a variable\n> declared via `local`. To assert that the command is actually successful\n> we also chain it with `&&`. This construct is seemingly not portable and\n> may fail with \"local: 1: bad variable name\".\n>\n> Split up the variable declaration and assignment to fix this.\n\nUnder what configuration, circumstances, conditions did you encounter\nthis problem? I ask because this isn't the only such case in the test\nsuite, as shown by:\n\n git grep -nP 'local ..*=\\$\\(..*&&' -- t\n\nWhat makes this case distinct from the others? Including such\ninformation would improve the commit message and help future readers.\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\n> @@ -80,8 +80,9 @@ test_expect_success 'init: reinitializing reftable with files backend fails' '\n> test_expect_perms () {\n> local perms=\"$1\"\n> local file=\"$2\"\n> - local actual=$(ls -l \"$file\") &&\n> + local actual\n>\n> + actual=$(ls -l \"$file\") &&\n"},{"id":"492286","messageId":"Zg_HFtiami4j5th7@ncase","threadId":"61266","inReplyTo":"CAPig+cSb_cKXYaNCBpe9Uy_iGB_K2NXaw4d1hg5bPuVaCEWvjA@mail.gmail.com","subject":"Re: [PATCH 11/12] t0610: fix non-portable variable assignment","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-05T09:40:38Z","receivedAt":"2024-04-05T09:40:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Apr 05, 2024 at 05:14:34AM -0400, Eric Sunshine wrote:\n> On Fri, Apr 5, 2024 at 3:51 AM Patrick Steinhardt wrote:\n> > In `test_expect_perms()` we assign the output of a command to a variable\n> > declared via `local`. To assert that the command is actually successful\n> > we also chain it with `&&`. This construct is seemingly not portable and\n> > may fail with \"local: 1: bad variable name\".\n> >\n> > Split up the variable declaration and assignment to fix this.\n> \n> Under what configuration, circumstances, conditions did you encounter\n> this problem? I ask because this isn't the only such case in the test\n> suite, as shown by:\n> \n> git grep -nP 'local ..*=\\$\\(..*&&' -- t\n> \n> What makes this case distinct from the others? Including such\n> information would improve the commit message and help future readers.\n\nI only ever saw this in some subset of GitHub CI jobs. See e.g. [1],\nwhere the logs [2] show the following error:\n\n```\nok 6 - init: reinitializing reftable with files backend fails\n+ test_when_finished rm -rf repo\n+ test 0 = 0\n+ test_cleanup={ rm -rf repo\n } && (exit \"$eval_ret\"); eval_ret=$?; :\n+ umask 002\n+ git init --shared=true repo\nInitialized empty shared Git repository in /home/runner/work/git/git/t/trash directory.t0610-reftable-basics/repo/.git/\n+ git -C repo config core.sharedrepository\n+ test 1 = 1\n+ test_expect_perms -rw-rw-r-- repo/.git/reftable/tables.list\n+ local perms=-rw-rw-r--\n+ local file=repo/.git/reftable/tables.list\n+ ls -l repo/.git/reftable/tables.list\n+ local actual=-rw-rw-r-- 1 runner docker 43 Apr 4 11:55 repo/.git/reftable/tables.list\nt0610-reftable-basics.sh: 83: local: 1: bad variable name\n+ die\n+ code=2\n+ test_atexit_handler\n+ test : != :\n+ return 0\n+ test -n\n+ echo FATAL: Unexpected exit with code 2\nFATAL: Unexpected exit with code 2\n```\n\nI was a bit surprised by this, too, and cannot reproduce it with any of\nmy systems. But I would bet this is dependent on the actual version of\nyour shell because it only happened with the Ubuntu 20.04 and 16.04\njobs.\n\nI didn't dig much deeper though as the fix is easy enough, even though\nit is surprising.\n\n[1]: https://github.com/git/git/actions/runs/8554157462/job/23438877643\n[2]: https://github.com/git/git/actions/runs/8554157462/artifacts/1384620962\n\nPatrick\n\n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\n> > @@ -80,8 +80,9 @@ test_expect_success 'init: reinitializing reftable with files backend fails' '\n> > test_expect_perms () {\n> > local perms=\"$1\"\n> > local file=\"$2\"\n> > - local actual=$(ls -l \"$file\") &&\n> > + local actual\n> >\n> > + actual=$(ls -l \"$file\") &&\n"},{"id":"492287","messageId":"Zg_HPQFlNazTlDRa@ncase","threadId":"61266","inReplyTo":"CAOw_e7b5wDKOcY9hSty+c+z-NWVUgri3h3ScF0qd1Y6cmfsyVA@mail.gmail.com","subject":"Re: [PATCH 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-05T09:41:17Z","receivedAt":"2024-04-05T09:41:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 04, 2024 at 11:40:29PM +0200, Han-Wen Nienhuys wrote:\n> On Thu, Apr 4, 2024 at 5:01 PM Patrick Steinhardt wrote:\n> > +\n> > +test_same_refs () {\n> > + git show-ref --head >cgit.actual &&\n> > + jgit show-ref >jgit-tabs.actual &&\n> \n> This seeks to the start and then iterates to the end, likely skipping\n> indexes. If you want to test for indexes etc. you should also sample\n> random refs from the namespace and see if they are returned correctly.\n\nTrue, I'll add that.\n\n> > +test_expect_success 'JGit repository can be read by CGit' '\n> > + test_when_finished \"rm -rf repo\" &&\n> > + # JGit does not provide a way to create a reftable-enabled repository.\n> \n> You can do \"jgit init && jgit convert-refstorage --format=reftable\"\n\nPerfect, thanks for this hint!\n\nPatrick\n"},{"id":"492294","messageId":"xmqqmsq7yezc.fsf@gitster.g","threadId":"61266","inReplyTo":"c2c2747ff57f68ccad8b509af037e1fc4a524fa1.1712235356.git.ps@pks.im","subject":"Re: [PATCH 11/12] t0610: fix non-portable variable assignment","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-05T16:02:47Z","receivedAt":"2024-04-05T16:02:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> test_expect_perms () {\n> \tlocal perms=\"$1\"\n> \tlocal file=\"$2\"\n> -\tlocal actual=$(ls -l \"$file\") &&\n> +\tlocal actual\n> \n> +\tactual=$(ls -l \"$file\") &&\n\nIsn't this the same as what ebee5580 (parallel-checkout: avoid dash\nlocal bug in tests, 2021-06-06) fixed?\n\nThe rule for variable assignment is mishandled when local is\ninvolved by some shells.\n\n\tperms=$1\n\tfile=$2\n\tactual=$(ls -l \"$file\")\n\nwould be perfectly fine, and should be fine with \"local\" prefixed on\nthese lines, but the last one with local without \"\" qround $(...)\nincorrectly makes the substitution subject to field splitting.\n\nI think the right fix should look rather like this, instead.\n\n t/t0610-reftable-basics.sh | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git i/t/t0610-reftable-basics.sh w/t/t0610-reftable-basics.sh\nindex 686781192e..894896933e 100755\n--- i/t/t0610-reftable-basics.sh\n+++ w/t/t0610-reftable-basics.sh\n@@ -81,9 +81,9 @@ test_expect_success 'init: reinitializing reftable with files backend fails' '\n '\n \n test_expect_perms () {\n-\tlocal perms=\"$1\"\n-\tlocal file=\"$2\"\n-\tlocal actual=$(ls -l \"$file\") &&\n+\tlocal perms=\"$1\" &&\n+\tlocal file=\"$2\" &&\n+\tlocal actual=\"$(ls -l \"$file\")\" &&\n \n \tcase \"$actual\" in\n \t$perms*)\n\n"},{"id":"492295","messageId":"xmqqbk6nyej1.fsf_-_@gitster.g","threadId":"61266","inReplyTo":"xmqqmsq7yezc.fsf@gitster.g","subject":"[PATCH] CodingGuidelines: quote assigned value with \"local\" and \"export\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-05T16:12:34Z","receivedAt":"2024-04-05T16:12:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Dash bug https://bugs.launchpad.net/ubuntu/+source/dash/+bug/139097\nlets the shell erroneously perform field splitting on the expansion\nof a command substitution during declaration of a local or an extern\nvariable.\n\nThe explanation was stolen from ebee5580 (parallel-checkout: avoid\ndash local bug in tests, 2021-06-06).\n\nSigned-off-by: Junio C Hamano \n---\n\n * The coding guidelines now forbids use of local/export var=$val\n without having $val quoted inside a pair of double quotes to\n avoid portability bugs.\n\n> Isn't this the same as what ebee5580 (parallel-checkout: avoid dash\n> local bug in tests, 2021-06-06) fixed?\n\n Documentation/CodingGuidelines | 14 ++++++++++++++\n 1 file changed, 14 insertions(+)\n\ndiff --git i/Documentation/CodingGuidelines w/Documentation/CodingGuidelines\nindex 32e69f798e..8aa76094ca 100644\n--- i/Documentation/CodingGuidelines\n+++ w/Documentation/CodingGuidelines\n@@ -188,6 +188,20 @@ For shell scripts specifically (not exhaustive):\n hopefully nobody starts using \"local\" before they are reimplemented\n in C ;-)\n \n+ - Some versions of dash has broken variable assignment when prefixed\n+ with \"local\", \"export\", and \"readonly\", in that the value to be\n+ assigned goes through field splitting at $IFS unless quoted. \n+\n+ DO NOT write:\n+\n+ local variable=$value ;# wrong\n+ export variable=$(command args) ;# wrong\n+\n+ and instead write:\n+\n+ local variable=\"$value\"\n+ export variable=\"$(command args)\"\n+\n - Use octal escape sequences (e.g. \"\\302\\242\"), not hexadecimal (e.g.\n \"\\xc2\\xa2\") in printf format strings, since hexadecimal escape\n sequences are not portable.\n"},{"id":"492296","messageId":"xmqqwmpbwzje.fsf@gitster.g","threadId":"61266","inReplyTo":"xmqqbk6nyej1.fsf_-_@gitster.g","subject":"Re: [PATCH] CodingGuidelines: quote assigned value with \"local\" and \"export\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-05T16:21:41Z","receivedAt":"2024-04-05T16:21:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Somebody may want to inspect the output from\n\n $ git grep -E -i -e '^\t*(local|export) [a-z0-9_]*=\\$'\n\nand fix those that can be used with dash. I made a cursory scan and\nremoved obviously safe ones whose RHS will never have $IFS whitespaces,\nand the following are remainders. #leftoverbits\n\nt/t0610-reftable-basics.sh:\tlocal actual=$(ls -l \"$file\") &&\nt/test-lib-functions.sh:\tlocal file=${2:-\"$1.t\"} &&\nt/test-lib-functions.sh:\tlocal basename=${1#??}\nt/test-lib-functions.sh:\tlocal var=$1 port\nt/test-lib-functions.sh:\tlocal expr=$(printf '\"%s\",' \"$@\")\nt/test-lib-functions.sh:\tlocal inc=${2:-0} &&\nt/test-lib-functions.sh:\tlocal inc=${2:-0} &&\nt/test-lib-functions.sh:\tlocal ret=$?\n"},{"id":"492298","messageId":"ZhAqamxWZNlMKxWJ@ncase","threadId":"61266","inReplyTo":"xmqqmsq7yezc.fsf@gitster.g","subject":"Re: [PATCH 11/12] t0610: fix non-portable variable assignment","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-05T16:44:26Z","receivedAt":"2024-04-05T16:44:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Apr 05, 2024 at 09:02:47AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > test_expect_perms () {\n> > \tlocal perms=\"$1\"\n> > \tlocal file=\"$2\"\n> > -\tlocal actual=$(ls -l \"$file\") &&\n> > +\tlocal actual\n> > \n> > +\tactual=$(ls -l \"$file\") &&\n> \n> Isn't this the same as what ebee5580 (parallel-checkout: avoid dash\n> local bug in tests, 2021-06-06) fixed?\n> \n> The rule for variable assignment is mishandled when local is\n> involved by some shells.\n> \n> \tperms=$1\n> \tfile=$2\n> \tactual=$(ls -l \"$file\")\n> \n> would be perfectly fine, and should be fine with \"local\" prefixed on\n> these lines, but the last one with local without \"\" qround $(...)\n> incorrectly makes the substitution subject to field splitting.\n> \n> I think the right fix should look rather like this, instead.\n\nAh, interesting, thanks for the pointer! I'll send v2 of this series\nearly next week.\n\nPatrick\n\n> t/t0610-reftable-basics.sh | 6 +++---\n> 1 file changed, 3 insertions(+), 3 deletions(-)\n> \n> diff --git i/t/t0610-reftable-basics.sh w/t/t0610-reftable-basics.sh\n> index 686781192e..894896933e 100755\n> --- i/t/t0610-reftable-basics.sh\n> +++ w/t/t0610-reftable-basics.sh\n> @@ -81,9 +81,9 @@ test_expect_success 'init: reinitializing reftable with files backend fails' '\n> '\n> \n> test_expect_perms () {\n> -\tlocal perms=\"$1\"\n> -\tlocal file=\"$2\"\n> -\tlocal actual=$(ls -l \"$file\") &&\n> +\tlocal perms=\"$1\" &&\n> +\tlocal file=\"$2\" &&\n> +\tlocal actual=\"$(ls -l \"$file\")\" &&\n> \n> \tcase \"$actual\" in\n> \t$perms*)\n> \n"},{"id":"492313","messageId":"20240405174859.GE2529133@coredump.intra.peff.net","threadId":"61266","inReplyTo":"xmqqbk6nyej1.fsf_-_@gitster.g","subject":"Re: [PATCH] CodingGuidelines: quote assigned value with \"local\" and \"export\"","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-04-05T17:48:59Z","receivedAt":"2024-04-05T17:49:00Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 05, 2024 at 09:12:34AM -0700, Junio C Hamano wrote:\n\n> Dash bug https://bugs.launchpad.net/ubuntu/+source/dash/+bug/139097\n> lets the shell erroneously perform field splitting on the expansion\n> of a command substitution during declaration of a local or an extern\n> variable.\n> \n> The explanation was stolen from ebee5580 (parallel-checkout: avoid\n> dash local bug in tests, 2021-06-06).\n\nThanks for digging up that commit. I read the earlier part of the thread\nand went off on a wild goose chase in the archive. :)\n\n> + - Some versions of dash has broken variable assignment when prefixed\n> + with \"local\", \"export\", and \"readonly\", in that the value to be\n> + assigned goes through field splitting at $IFS unless quoted. \n> +\n> + DO NOT write:\n> +\n> + local variable=$value ;# wrong\n> + export variable=$(command args) ;# wrong\n> +\n> + and instead write:\n> +\n> + local variable=\"$value\"\n> + export variable=\"$(command args)\"\n\nI think that is a good rule for \"local\", but I thought we did not allow\n\"export foo=bar\" at all, and required:\n\n foo=bar\n export foo\n\nIf that was only because of this bug, it would be nice to loosen the\nrules a bit.\n\n-Peff\n"},{"id":"492318","messageId":"xmqqr0fjtxei.fsf@gitster.g","threadId":"61266","inReplyTo":"20240405174859.GE2529133@coredump.intra.peff.net","subject":"Re: [PATCH] CodingGuidelines: quote assigned value with \"local\" and \"export\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-05T19:36:05Z","receivedAt":"2024-04-05T19:36:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> + - Some versions of dash has broken variable assignment when prefixed\n>> + with \"local\", \"export\", and \"readonly\", in that the value to be\n>> + assigned goes through field splitting at $IFS unless quoted. \n>> +\n>> + DO NOT write:\n>> +\n>> + local variable=$value ;# wrong\n>> + export variable=$(command args) ;# wrong\n>> +\n>> + and instead write:\n>> +\n>> + local variable=\"$value\"\n>> + export variable=\"$(command args)\"\n>\n> I think that is a good rule for \"local\", but I thought we did not allow\n> \"export foo=bar\" at all, and required:\n>\n> foo=bar\n> export foo\n>\n> If that was only because of this bug, it would be nice to loosen the\n> rules a bit.\n\nThat rule in Documentation/CodingGuidelines predates the discovery\nof this bug. I have this vague feeling that it was for the shell on\nold Solaris, which would not matter to us anymore, but I do not\nremember.\n\nAs we are not showing \"readonly\" in the \"DO NOT/DO\" example above,\nwe should probably drop the \"export\" example and discuss it\nseparately and decide if it makes sense to loosen the \"export var\"\nvs \"export var=val\" rule.\n\nThanks.\n"},{"id":"492319","messageId":"ZhBX7REqotKKHpWg@google.com","threadId":"61266","inReplyTo":"22f86f8ccb9f3fb7f98ff57ddd09724fc9e44628.1712235356.git.ps@pks.im","subject":"Re: [PATCH 08/12] ci: make Perforce binaries executable for all users","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2024-04-05T20:01:52Z","receivedAt":"2024-04-05T20:02:00Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2024.04.04 15:25, Patrick Steinhardt wrote:\n> The Perforce binaries are only made executable for the current user. On\n> GitLab CI though we execute tests as a different user than \"root\", and\n> thus these binaries may not be executable by that test user.\n> \n> Fix the setup so that we set the executable bits for all users.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> ci/install-dependencies.sh | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> index f4eb125fd2..068c478025 100755\n> --- a/ci/install-dependencies.sh\n> +++ b/ci/install-dependencies.sh\n> @@ -47,7 +47,7 @@ ubuntu-*)\n> \tmkdir --parents \"$CUSTOM_PATH\"\n> \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n> \t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n> -\tchmod u+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n> +\tchmod a+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n> \n> \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n> \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n> -- \n> 2.44.GIT\n> \n\nDo we break CI in patch 6 and 7 until we get this fix? Perhaps we should\njust squash this into patch 6?\n"},{"id":"492330","messageId":"xmqq4jcftn82.fsf@gitster.g","threadId":"61266","inReplyTo":"xmqqr0fjtxei.fsf@gitster.g","subject":"Re: [PATCH] CodingGuidelines: quote assigned value with \"local\" and \"export\"","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-05T23:15:57Z","receivedAt":"2024-04-05T23:16:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> I think that is a good rule for \"local\", but I thought we did not allow\n>> \"export foo=bar\" at all, and required:\n>>\n>> foo=bar\n>> export foo\n>>\n>> If that was only because of this bug, it would be nice to loosen the\n>> rules a bit.\n>\n> That rule in Documentation/CodingGuidelines predates the discovery\n> of this bug. I have this vague feeling that it was for the shell on\n> old Solaris, which would not matter to us anymore, but I do not\n> remember.\n\nHeh, I do not even see any such rule in the guidelines. What\nenforces it is actually in t/check-non-portable-shell.pl script. It\ncame from 9968ffff (test-lint: detect 'export FOO=bar', 2013-07-08),\nwhich in turn comes from https://lore.kernel.org/git/201307081121.22769.tboegi@web.de/\n\nWe make multiple uses of it in ci/*.sh but the environments ci/\nscripts are used in are rather sterile, so they do not quite count\nas a proof that the problematic shells no longer exist.\n\nWe may instead want to add a separate rule e.g.,\n\n\t/\\blocal\\s+[a-zA-z0-9_]*=\\$/ and err q(quote \"$val\" in 'local var=$val');\n\nto the check script.\n\n\n"},{"id":"492382","messageId":"CAOw_e7bx=KEeA6sdoDPti6LvxU9Pmr9ZyzyrXfyTmT3DUcHL6w@mail.gmail.com","threadId":"61266","inReplyTo":"Zg_HPQFlNazTlDRa@ncase","subject":"Re: [PATCH 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Han-Wen Nienhuys","fromEmail":"hanwenn@gmail.com","sentAt":"2024-04-06T13:03:20Z","receivedAt":"2024-04-06T13:03:32Z","isPatch":true,"sender":{"key":"hanwenn@gmail.com","avatar":"https://gravatar.com/avatar/058832acb8d613baeb6ce9d21b009d9772424a309b9a521330423daef909a27a?d=mp&s=160"},"body":"On Fri, Apr 5, 2024 at 11:41 AM Patrick Steinhardt wrote:\n> > > +test_expect_success 'JGit repository can be read by CGit' '\n> > > + test_when_finished \"rm -rf repo\" &&\n> > > + # JGit does not provide a way to create a reftable-enabled repository.\n> >\n> > You can do \"jgit init && jgit convert-refstorage --format=reftable\"\n>\n> Perfect, thanks for this hint!\n\non a tangent: I wrote this a long time ago, and it does no locking,\nwhich probably leads to unpleasant surprises if it is run in parallel\nwith other ref updates. What would be a foolproof way to stop other\nprocesses from stomping over the repo while the conversion is ongoing?\nMaybe create a packed-refs.lock file?\n\n\n-- \nHan-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen\n"},{"id":"492392","messageId":"ZhFwBJ3UqBJsq8xh@ncase","threadId":"61266","inReplyTo":"CAOw_e7bx=KEeA6sdoDPti6LvxU9Pmr9ZyzyrXfyTmT3DUcHL6w@mail.gmail.com","subject":"Re: [PATCH 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-06T15:53:40Z","receivedAt":"2024-04-06T15:53:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Apr 06, 2024 at 03:03:20PM +0200, Han-Wen Nienhuys wrote:\n> On Fri, Apr 5, 2024 at 11:41 AM Patrick Steinhardt wrote:\n> > > > +test_expect_success 'JGit repository can be read by CGit' '\n> > > > + test_when_finished \"rm -rf repo\" &&\n> > > > + # JGit does not provide a way to create a reftable-enabled repository.\n> > >\n> > > You can do \"jgit init && jgit convert-refstorage --format=reftable\"\n> >\n> > Perfect, thanks for this hint!\n> \n> on a tangent: I wrote this a long time ago, and it does no locking,\n> which probably leads to unpleasant surprises if it is run in parallel\n> with other ref updates. What would be a foolproof way to stop other\n> processes from stomping over the repo while the conversion is ongoing?\n> Maybe create a packed-refs.lock file?\n\nYes, this is indeed a problem that I have been thinking about quite a\nlot recently as I want to introduce migration tooling in the next\nrelease cycle. I was quite happy to learn that you've got something in\nJGit and was hoping that you found a way to avoid races. Too bad you\nseem to have the same issue as I do right now.\n\nIn any case, creating a \"packed-refs.lock\" is not sufficient. While it\ndoes protect against ref deletion and repacking refs, it doesn't protect\nagainst ref creation and updates. And in fact there is no way to guard\nagainst these at all as all that the \"files\" backend does is to create\nor update a single loose file without taking any central lock.\n\nI was briefly thinking about a \"migration\" repository extension. Once\nthe migration starts we add \"extensions.ref-migration\" to the repo,\nwhich is an extension that no client knows how to handle. Consequently,\nwhile the extension is set, any _new_ Git processes would bail out and\nrefuse to update the repository.\n\nThat only solves part of the problem though. What it doesn't protect\nagainst is already-running Git processes that have already read the\nconfig.\n\nThere are of course very hacky ways to get closer to the goal. E.g.:\n\n 1. Repack all refs.\n\n 2. Delete the now-empty \"refs/\" hierarchy.\n\n 3. Create a file \"refs\" with invalid contents.\n\n 4. Lock the \"packed-refs\" file.\n\n 5. Lock the \"HEAD\" file.\n\nNow you can be sure that there is no way to create or update normal\nrefs. But it's still incomplete as pseudo-refs can be written to. And it\ndoes feel very fragile overall, especially if you need to roll back the\nmigration due to whatever reason.\n\nSo I dunno, until now I didn't yet find any way to reliably avoid racy\nref updates during a conversion. If you or anybody else got some ideas\nI'd be very happy to hear about them.\n\nPatrick\n"},{"id":"492423","messageId":"20240407012357.GB1085004@coredump.intra.peff.net","threadId":"61266","inReplyTo":"xmqq4jcftn82.fsf@gitster.g","subject":"Re: [PATCH] CodingGuidelines: quote assigned value with \"local\" and \"export\"","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-04-07T01:23:57Z","receivedAt":"2024-04-07T01:23:59Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 05, 2024 at 04:15:57PM -0700, Junio C Hamano wrote:\n\n> Junio C Hamano writes:\n> \n> >> I think that is a good rule for \"local\", but I thought we did not allow\n> >> \"export foo=bar\" at all, and required:\n> >>\n> >> foo=bar\n> >> export foo\n> >>\n> >> If that was only because of this bug, it would be nice to loosen the\n> >> rules a bit.\n> >\n> > That rule in Documentation/CodingGuidelines predates the discovery\n> > of this bug. I have this vague feeling that it was for the shell on\n> > old Solaris, which would not matter to us anymore, but I do not\n> > remember.\n> \n> Heh, I do not even see any such rule in the guidelines. What\n> enforces it is actually in t/check-non-portable-shell.pl script. It\n> came from 9968ffff (test-lint: detect 'export FOO=bar', 2013-07-08),\n> which in turn comes from https://lore.kernel.org/git/201307081121.22769.tboegi@web.de/\n\nYeah, I noticed it was not in CodingGuidelines, but did not even realize\nit was in the linter. Thanks for digging up the link, though sadly it\ndoes not say which shell had a problem. I could very well believe there\nis no such modern shell, but I don't know how to test that without a\nweather balloon patch.\n\n> We make multiple uses of it in ci/*.sh but the environments ci/\n> scripts are used in are rather sterile, so they do not quite count\n> as a proof that the problematic shells no longer exist.\n> \n> We may instead want to add a separate rule e.g.,\n> \n> \t/\\blocal\\s+[a-zA-z0-9_]*=\\$/ and err q(quote \"$val\" in 'local var=$val');\n> \n> to the check script.\n\nYeah. I think matching \\$ is probably enough to catch most relevant\ncases without complaining about the innocuous:\n\n local foo=bar\n\nIt would miss:\n\n local foo=\"bar/$1\"\n\nI guess \"=[^\"]*\\$\" would be a bit more aggressive.\n\n-Peff\n"},{"id":"492483","messageId":"ZhN_jwNAcR7UhCc3@tanuki","threadId":"61266","inReplyTo":"CAOw_e7b5wDKOcY9hSty+c+z-NWVUgri3h3ScF0qd1Y6cmfsyVA@mail.gmail.com","subject":"Re: [PATCH 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T05:24:31Z","receivedAt":"2024-04-08T05:24:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 04, 2024 at 11:40:29PM +0200, Han-Wen Nienhuys wrote:\n> On Thu, Apr 4, 2024 at 5:01 PM Patrick Steinhardt wrote:\n> > +test_expect_success 'JGit repository can be read by CGit' '\n> > + test_when_finished \"rm -rf repo\" &&\n> > + # JGit does not provide a way to create a reftable-enabled repository.\n> \n> You can do \"jgit init && jgit convert-refstorage --format=reftable\"\n\nBy the way, the above command does not work as-is as it will end up with\na repository that has no \"HEAD\" reference. I can work around that\nthough by first writing the default branch and then converting the ref\nstorage.\n\nPatrick\n"},{"id":"492487","messageId":"ZhOFLsfVg-dpQJ4w@tanuki","threadId":"61266","inReplyTo":"ZhBX7REqotKKHpWg@google.com","subject":"Re: [PATCH 08/12] ci: make Perforce binaries executable for all users","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T05:48:30Z","receivedAt":"2024-04-08T05:48:36Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Apr 05, 2024 at 01:01:52PM -0700, Josh Steadmon wrote:\n> On 2024.04.04 15:25, Patrick Steinhardt wrote:\n> > The Perforce binaries are only made executable for the current user. On\n> > GitLab CI though we execute tests as a different user than \"root\", and\n> > thus these binaries may not be executable by that test user.\n> > \n> > Fix the setup so that we set the executable bits for all users.\n> > \n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > ci/install-dependencies.sh | 2 +-\n> > 1 file changed, 1 insertion(+), 1 deletion(-)\n> > \n> > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> > index f4eb125fd2..068c478025 100755\n> > --- a/ci/install-dependencies.sh\n> > +++ b/ci/install-dependencies.sh\n> > @@ -47,7 +47,7 @@ ubuntu-*)\n> > \tmkdir --parents \"$CUSTOM_PATH\"\n> > \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n> > \t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n> > -\tchmod u+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n> > +\tchmod a+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n> > \n> > \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n> > \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n> > -- \n> > 2.44.GIT\n> > \n> \n> Do we break CI in patch 6 and 7 until we get this fix? Perhaps we should\n> just squash this into patch 6?\n\nNo, this was broken before already as we had the same \"chmod u+x\" even\nwithout this patch series. This doesn't lead to a broken CI system\nthough as Perforce is an optional dependency. Instead it causes us to\nskip all Perforce tests because we won't be able to look up these\nbinaries via PATH.\n\nI'll try to clarify the commit message.\n\nPatrick\n"},{"id":"492492","messageId":"cover.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH v2 00/12] t: exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:45:36Z","receivedAt":"2024-04-08T06:45:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that adds compatibility\ntests for reftables to check whether the Git and JGit implementations\nare compatible with each other.\n\nChanges compared to v1:\n\n - Patch 8: Clarified that this has been broken for a rather long\n time, but that it was \"silently\" broken.\n\n - Patch 11: adapt the fix for the non-portable \"local\" variable\n assignment based on the discussion.\n\n - Patch 12: extend tests to do some basic ref comparisons, which\n should exercise indices more thoroughly.\n\nCI runs for this series:\n\n - https://github.com/git/git/actions/runs/8595241646/\n - https://gitlab.com/gitlab-org/git/-/pipelines/1243766428\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (12):\n ci: rename \"runs_on_pool\" to \"distro\"\n ci: expose distro name in dockerized GitHub jobs\n ci: allow skipping sudo on dockerized jobs\n ci: drop duplicate package installation for \"linux-gcc-default\"\n ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"\n ci: merge custom PATH directories\n ci: merge scripts which install dependencies\n ci: make Perforce binaries executable for all users\n ci: install JGit dependency\n t06xx: always execute backend-specific tests\n t0610: fix non-portable variable assignment\n t0612: add tests to exercise Git/JGit reftable compatibility\n\n .github/workflows/main.yml | 8 +-\n .gitlab-ci.yml | 4 +-\n ci/install-dependencies.sh | 100 +++++++++++++------\n ci/install-docker-dependencies.sh | 46 ---------\n ci/lib.sh | 14 +--\n t/t0600-reffiles-backend.sh | 8 +-\n t/t0601-reffiles-pack-refs.sh | 9 +-\n t/t0610-reftable-basics.sh | 15 ++-\n t/t0612-reftable-jgit-compatibility.sh | 132 +++++++++++++++++++++++++\n 9 files changed, 226 insertions(+), 110 deletions(-)\n delete mode 100755 ci/install-docker-dependencies.sh\n create mode 100755 t/t0612-reftable-jgit-compatibility.sh\n\nRange-diff against v1:\n 1: e618129549 = 1: 89723b6812 ci: rename \"runs_on_pool\" to \"distro\"\n 2: e3e2b7cd50 = 2: e60a40bd65 ci: expose distro name in dockerized GitHub jobs\n 3: 8abc9ad6a7 = 3: 16603d40fd ci: allow skipping sudo on dockerized jobs\n 4: 7cf2538625 = 4: b4f6d6d3bf ci: drop duplicate package installation for \"linux-gcc-default\"\n 5: 38e64224e2 = 5: 6abc53bf51 ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"\n 6: 196dab460a = 6: d9be4db56f ci: merge custom PATH directories\n 7: 668553e18f = 7: 4a90c003d1 ci: merge scripts which install dependencies\n 8: 22f86f8ccb ! 8: 5240046a0f ci: make Perforce binaries executable for all users\n @@ Commit message\n \n The Perforce binaries are only made executable for the current user. On\n GitLab CI though we execute tests as a different user than \"root\", and\n - thus these binaries may not be executable by that test user.\n + thus these binaries may not be executable by that test user at all. This\n + has gone unnoticed so far because those binaries are optional -- in case\n + they don't exist we simply skip over tests requiring them.\n \n Fix the setup so that we set the executable bits for all users.\n \n 9: 1deded615e = 9: 29ceb623b9 ci: install JGit dependency\n10: 51c45c879f = 10: fc3472cdf3 t06xx: always execute backend-specific tests\n11: c2c2747ff5 ! 11: cedf5929d1 t0610: fix non-portable variable assignment\n @@ Metadata\n ## Commit message ##\n t0610: fix non-portable variable assignment\n \n - In `test_expect_perms()` we assign the output of a command to a variable\n - declared via `local`. To assert that the command is actually successful\n - we also chain it with `&&`. This construct is seemingly not portable and\n - may fail with \"local: 1: bad variable name\".\n + Older versions of the Dash shell fail to parse `local var=val`\n + assignments in some cases when `val` is unquoted. Such failures can be\n + observed e.g. with Ubuntu 20.04 and older, which has a Dash version that\n + still has this bug.\n \n - Split up the variable declaration and assignment to fix this.\n + Such an assignment has been introduced in t0610. The issue wasn't\n + detected for a while because this test used to only run when the\n + GIT_TEST_DEFAULT_REF_FORMAT environment variable was set to \"refatble\".\n + We have dropped that requirement now though, meaning that it runs\n + unconditionally, inclluding on jobs which use such older versions of\n + Ubuntu.\n +\n + We have worked around such issues in the past, e.g. in ebee5580ca\n + (parallel-checkout: avoid dash local bug in tests, 2021-06-06), by\n + quoting the `val` side. Apply the same fix to t0610.\n \n Signed-off-by: Patrick Steinhardt \n \n ## t/t0610-reftable-basics.sh ##\n @@ t/t0610-reftable-basics.sh: test_expect_success 'init: reinitializing reftable with files backend fails' '\n + '\n + \n test_expect_perms () {\n - \tlocal perms=\"$1\"\n - \tlocal file=\"$2\"\n +-\tlocal perms=\"$1\"\n +-\tlocal file=\"$2\"\n -\tlocal actual=$(ls -l \"$file\") &&\n -+\tlocal actual\n ++\tlocal perms=\"$1\" &&\n ++\tlocal file=\"$2\" &&\n ++\tlocal actual=\"$(ls -l \"$file\")\" &&\n \n -+\tactual=$(ls -l \"$file\") &&\n \tcase \"$actual\" in\n \t$perms*)\n - \t\t: happy\n12: db66dd4155 ! 12: 160b026e69 t0612: add tests to exercise Git/JGit reftable compatibility\n @@ t/t0612-reftable-jgit-compatibility.sh (new)\n +\ttest_cmp cgit.actual jgit.actual\n +}\n +\n ++test_same_ref () {\n ++\tgit rev-parse \"$1\" >cgit.actual &&\n ++\tjgit rev-parse \"$1\" >jgit.actual &&\n ++\ttest_cmp cgit.actual jgit.actual\n ++}\n ++\n +test_same_reflog () {\n +\tgit reflog \"$*\" >cgit.actual &&\n +\tjgit reflog \"$*\" >jgit-newline.actual &&\n @@ t/t0612-reftable-jgit-compatibility.sh (new)\n +\t\tcd repo &&\n +\t\ttest_commit A &&\n +\t\ttest_same_refs &&\n ++\t\ttest_same_ref HEAD &&\n +\t\ttest_same_reflog HEAD\n +\t)\n +'\n +\n +test_expect_success 'JGit repository can be read by CGit' '\n +\ttest_when_finished \"rm -rf repo\" &&\n -+\t# JGit does not provide a way to create a reftable-enabled repository.\n -+\tgit init repo &&\n ++\tjgit init repo &&\n +\t(\n +\t\tcd repo &&\n ++\n +\t\ttouch file &&\n +\t\tjgit add file &&\n +\t\tjgit commit -m \"initial commit\" &&\n +\n ++\t\t# Note that we must convert the ref storage after we have\n ++\t\t# written the default branch. Otherwise JGit will end up with\n ++\t\t# no HEAD at all.\n ++\t\tjgit convert-ref-storage --format=reftable &&\n ++\n +\t\ttest_same_refs &&\n ++\t\ttest_same_ref HEAD &&\n +\t\t# Interestingly, JGit cannot read its own reflog here. CGit can\n +\t\t# though.\n +\t\tprintf \"%s HEAD@{0}: commit (initial): initial commit\" \"$(git rev-parse --short HEAD)\" >expect &&\n @@ t/t0612-reftable-jgit-compatibility.sh (new)\n +\t\ttest_commit_jgit D &&\n +\n +\t\ttest_same_refs &&\n ++\t\ttest_same_ref HEAD &&\n +\t\ttest_same_reflog HEAD\n +\t)\n +'\n @@ t/t0612-reftable-jgit-compatibility.sh (new)\n +\t\t\" >input &&\n +\t\tgit update-ref --stdin \n---\n .github/workflows/main.yml | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 684ef5c00d..71cd4e5486 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -342,12 +342,16 @@ jobs:\n vector:\n - jobname: linux-musl\n image: alpine\n+ distro: alpine-latest\n - jobname: linux32\n image: daald/ubuntu32:xenial\n+ distro: ubuntu32-16.04\n - jobname: pedantic\n image: fedora\n+ distro: fedora-latest\n env:\n jobname: ${{matrix.vector.jobname}}\n+ distro: ${{matrix.vector.distro}}\n runs-on: ubuntu-latest\n container: ${{matrix.vector.image}}\n steps:\n-- \n2.44.GIT\n\n"},{"id":"492494","messageId":"16603d40fdf96948580c04a7c2b791a97ec64fe7.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"[PATCH v2 03/12] ci: allow skipping sudo on dockerized jobs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:46:22Z","receivedAt":"2024-04-08T06:46:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our \"install-dependencies.sh\" script is executed by non-dockerized jobs\nto install dependencies. These jobs don't run with \"root\" permissions,\nbut with a separate user. Consequently, we need to use sudo(8) there to\nelevate permissions when installing packages.\n\nWe're about to merge \"install-docker-dependencies.sh\" into that script\nthough, and our Docker containers do run as \"root\". Using sudo(8) is\nthus unnecessary there, even though it would be harmless. On some images\nlike Alpine Linux though there is no sudo(8) available by default, which\nwould consequently break the build.\n\nAdapt the script to make \"sudo\" a no-op when running as \"root\" user.\nThis allows us to easily reuse the script for our dockerized jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7d247b5ef4..7dfd3e50ed 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -11,6 +11,17 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n+# Make sudo a no-op and execute the command directly when running as root.\n+# While using sudo would be fine on most platforms when we are root already,\n+# some platforms like e.g. Alpine Linux do not have sudo available by default\n+# and would thus break.\n+if test \"$(id -u)\" -eq 0\n+then\n+\tsudo () {\n+\t\t\"$@\"\n+\t}\n+fi\n+\n case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n-- \n2.44.GIT\n\n"},{"id":"492495","messageId":"b4f6d6d3bfe28164f74ac927a0ac67950827d7df.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"[PATCH v2 04/12] ci: drop duplicate package installation for \"linux-gcc-default\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:46:27Z","receivedAt":"2024-04-08T06:46:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"linux-gcc-default\" job installs common Ubuntu packages. This is\nalready done in the distro-specific switch, so we basically duplicate\nthe effort here.\n\nDrop the duplicate package installations and inline the variable that\ncontains those common packages.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 17 ++++++++---------\n 1 file changed, 8 insertions(+), 9 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7dfd3e50ed..fad53aac96 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -7,9 +7,6 @@\n \n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n-UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n- tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n- libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n # Make sudo a no-op and execute the command directly when running as root.\n # While using sudo would be fine on most platforms when we are root already,\n@@ -25,8 +22,13 @@ fi\n case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n-\tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n-\t\t$UBUNTU_COMMON_PKGS $CC_PACKAGE $PYTHON_PACKAGE\n+\tsudo apt-get -q -y install \\\n+\t\tlanguage-pack-is libsvn-perl apache2 \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev \\\n+\t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n+\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n+\t\t$CC_PACKAGE $PYTHON_PACKAGE\n+\n \tmkdir --parents \"$P4_PATH\"\n \tpushd \"$P4_PATH\"\n \t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n@@ -34,6 +36,7 @@ ubuntu-*)\n \t\tchmod u+x p4d\n \t\tchmod u+x p4\n \tpopd\n+\n \tmkdir --parents \"$GIT_LFS_PATH\"\n \tpushd \"$GIT_LFS_PATH\"\n \t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n@@ -83,10 +86,6 @@ Documentation)\n \ttest -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n \tsudo gem install --version 1.5.8 asciidoctor\n \t;;\n-linux-gcc-default)\n-\tsudo apt-get -q update\n-\tsudo apt-get -q -y install $UBUNTU_COMMON_PKGS\n-\t;;\n esac\n \n if type p4d >/dev/null 2>&1 && type p4 >/dev/null 2>&1\n-- \n2.44.GIT\n\n"},{"id":"492496","messageId":"6abc53bf5173e7de3fa271d175145c1672f17a88.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"[PATCH v2 05/12] ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:46:31Z","receivedAt":"2024-04-08T06:46:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to merge the \"install-docker-dependencies.sh\" script into\n\"install-dependencies.sh\". This will also move our Alpine-based jobs\nover to use the latter script. This script uses the Bash shell though,\nwhich is not available by default on Alpine Linux.\n\nRefactor \"install-dependencies.sh\" to use \"/bin/sh\" instead of Bash.\nThis requires us to get rid of the pushd/popd invocations, which are\nreplaced by some more elaborate commands that download or extract\nexecutables right to where they are needed.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 28 +++++++++++-----------------\n 1 file changed, 11 insertions(+), 17 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex fad53aac96..7bcccc96fd 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -1,4 +1,4 @@\n-#!/usr/bin/env bash\n+#!/bin/sh\n #\n # Install dependencies required to build and test Git on Linux and macOS\n #\n@@ -30,19 +30,14 @@ ubuntu-*)\n \t\t$CC_PACKAGE $PYTHON_PACKAGE\n \n \tmkdir --parents \"$P4_PATH\"\n-\tpushd \"$P4_PATH\"\n-\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n-\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\t\tchmod u+x p4d\n-\t\tchmod u+x p4\n-\tpopd\n+\twget --quiet --directory-prefix=\"$P4_PATH\" \\\n+\t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n+\tchmod u+x \"$P4_PATH/p4d\" \"$P4_PATH/p4\"\n \n \tmkdir --parents \"$GIT_LFS_PATH\"\n-\tpushd \"$GIT_LFS_PATH\"\n-\t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\t\ttar --extract --gunzip --file \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n-\tpopd\n+\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n+\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n+\trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n macos-*)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n@@ -53,11 +48,10 @@ macos-*)\n \tbrew link --force gettext\n \n \tmkdir -p \"$P4_PATH\"\n-\tpushd \"$P4_PATH\"\n-\t\twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n-\t\ttar -xf helix-core-server.tgz &&\n-\t\tsudo xattr -d com.apple.quarantine p4 p4d 2>/dev/null || true\n-\tpopd\n+\twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n+\ttar -xf helix-core-server.tgz -C \"$P4_PATH\" p4 p4d &&\n+\tsudo xattr -d com.apple.quarantine \"$P4_PATH/p4\" \"$P4_PATH/p4d\" 2>/dev/null || true\n+\trm helix-core-server.tgz\n \n \tif test -n \"$CC_PACKAGE\"\n \tthen\n-- \n2.44.GIT\n\n"},{"id":"492497","messageId":"d9be4db56fa028c96fb316413e87d3c37c3ba602.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"[PATCH v2 06/12] ci: merge custom PATH directories","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:46:36Z","receivedAt":"2024-04-08T06:46:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're downloading various executables required by our tests. Each of\nthese executables goes into its own directory, which is then appended to\nthe PATH variable. Consequently, whenever we add a new dependency and\nthus a new directory, we would have to adapt to this change in several\nplaces.\n\nRefactor this to instead put all binaries into a single directory.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 15 +++++++--------\n ci/lib.sh | 10 +++-------\n 2 files changed, 10 insertions(+), 15 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7bcccc96fd..46b9efb2d5 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -29,14 +29,13 @@ ubuntu-*)\n \t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n \t\t$CC_PACKAGE $PYTHON_PACKAGE\n \n-\tmkdir --parents \"$P4_PATH\"\n-\twget --quiet --directory-prefix=\"$P4_PATH\" \\\n+\tmkdir --parents \"$CUSTOM_PATH\"\n+\twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n \t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\tchmod u+x \"$P4_PATH/p4d\" \"$P4_PATH/p4\"\n+\tchmod u+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n \n-\tmkdir --parents \"$GIT_LFS_PATH\"\n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n+\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n macos-*)\n@@ -47,10 +46,10 @@ macos-*)\n \tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \n-\tmkdir -p \"$P4_PATH\"\n+\tmkdir -p \"$CUSTOM_PATH\"\n \twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n-\ttar -xf helix-core-server.tgz -C \"$P4_PATH\" p4 p4d &&\n-\tsudo xattr -d com.apple.quarantine \"$P4_PATH/p4\" \"$P4_PATH/p4d\" 2>/dev/null || true\n+\ttar -xf helix-core-server.tgz -C \"$CUSTOM_PATH\" p4 p4d &&\n+\tsudo xattr -d com.apple.quarantine \"$CUSTOM_PATH/p4\" \"$CUSTOM_PATH/p4d\" 2>/dev/null || true\n \trm helix-core-server.tgz\n \n \tif test -n \"$CC_PACKAGE\"\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex d882250db5..4cce854bad 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -340,10 +340,6 @@ ubuntu-*)\n \t# image.\n \t# Keep that in mind when you encounter a broken OS X build!\n \texport LINUX_GIT_LFS_VERSION=\"1.5.2\"\n-\n-\tP4_PATH=\"$HOME/custom/p4\"\n-\tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n-\texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n macos-*)\n \tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n@@ -351,12 +347,12 @@ macos-*)\n \tthen\n \t\tMAKEFLAGS=\"$MAKEFLAGS APPLE_COMMON_CRYPTO_SHA1=Yes\"\n \tfi\n-\n-\tP4_PATH=\"$HOME/custom/p4\"\n-\texport PATH=\"$P4_PATH:$PATH\"\n \t;;\n esac\n \n+CUSTOM_PATH=\"$HOME/path\"\n+export PATH=\"$CUSTOM_PATH:$PATH\"\n+\n case \"$jobname\" in\n linux32)\n \tCC=gcc\n-- \n2.44.GIT\n\n"},{"id":"492498","messageId":"4a90c003d1cdbf1e38efe5f7b78badf96a61b8d7.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"[PATCH v2 07/12] ci: merge scripts which install dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:46:41Z","receivedAt":"2024-04-08T06:46:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different scripts which install dependencies, one for\ndockerized jobs and one for non-dockerized ones. Naturally, these\nscripts have quite some duplication. Furthermore, either of these\nscripts is missing some test dependencies that the respective other\nscript has, thus reducing test coverage.\n\nMerge those two scripts such that there is a single source of truth for\ntest dependencies, only.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 2 +-\n .gitlab-ci.yml | 4 +--\n ci/install-dependencies.sh | 32 ++++++++++++++++++---\n ci/install-docker-dependencies.sh | 46 -------------------------------\n 4 files changed, 31 insertions(+), 53 deletions(-)\n delete mode 100755 ci/install-docker-dependencies.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 71cd4e5486..5838986895 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -359,7 +359,7 @@ jobs:\n if: matrix.vector.jobname != 'linux32'\n - uses: actions/checkout@v1 # cannot be upgraded because Node.js Actions aren't supported in this container\n if: matrix.vector.jobname == 'linux32'\n- - run: ci/install-docker-dependencies.sh\n+ - run: ci/install-dependencies.sh\n - run: ci/run-build-and-tests.sh\n - name: print test failures\n if: failure() && env.FAILED_TEST_ARTIFACTS != ''\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex c0fa2fe90b..8ce0e1b4bd 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -10,7 +10,7 @@ workflow:\n test:linux:\n image: $image\n before_script:\n- - ./ci/install-docker-dependencies.sh\n+ - ./ci/install-dependencies.sh\n script:\n - useradd builder --create-home\n - chown -R builder \"${CI_PROJECT_DIR}\"\n@@ -98,7 +98,7 @@ static-analysis:\n variables:\n jobname: StaticAnalysis\n before_script:\n- - ./ci/install-docker-dependencies.sh\n+ - ./ci/install-dependencies.sh\n script:\n - ./ci/run-static-analysis.sh\n - ./ci/check-directional-formatting.bash\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 46b9efb2d5..f4eb125fd2 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -5,6 +5,8 @@\n \n . ${0%/*}/lib.sh\n \n+begin_group \"Install dependencies\"\n+\n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n \n@@ -20,14 +22,27 @@ then\n fi\n \n case \"$distro\" in\n+alpine-*)\n+\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n+\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n+\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n+\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n+\t;;\n+fedora-*)\n+\tdnf -yq update >/dev/null &&\n+\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\t;;\n ubuntu-*)\n+\t# Required so that apt doesn't wait for user input on certain packages.\n+\texport DEBIAN_FRONTEND=noninteractive\n+\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install \\\n-\t\tlanguage-pack-is libsvn-perl apache2 \\\n-\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev \\\n+\t\tlanguage-pack-is libsvn-perl apache2 cvs cvsps git gnupg subversion \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo \\\n \t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n-\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n-\t\t$CC_PACKAGE $PYTHON_PACKAGE\n+\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \\\n+\t\t${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE\n \n \tmkdir --parents \"$CUSTOM_PATH\"\n \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n@@ -38,6 +53,13 @@ ubuntu-*)\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n+ubuntu32-*)\n+\tsudo linux32 --32bit i386 sh -c '\n+\t\tapt update >/dev/null &&\n+\t\tapt install -y build-essential libcurl4-openssl-dev \\\n+\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n+\t'\n+\t;;\n macos-*)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n \t# Uncomment this if you want to run perf tests:\n@@ -97,3 +119,5 @@ then\n else\n \techo >&2 \"WARNING: git-lfs wasn't installed, see above for clues why\"\n fi\n+\n+end_group \"Install dependencies\"\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\ndeleted file mode 100755\nindex eb2c9e1eca..0000000000\n--- a/ci/install-docker-dependencies.sh\n+++ /dev/null\n@@ -1,46 +0,0 @@\n-#!/bin/sh\n-#\n-# Install dependencies required to build and test Git inside container\n-#\n-\n-. ${0%/*}/lib.sh\n-\n-begin_group \"Install dependencies\"\n-\n-case \"$jobname\" in\n-linux32)\n-\tlinux32 --32bit i386 sh -c '\n-\t\tapt update >/dev/null &&\n-\t\tapt install -y build-essential libcurl4-openssl-dev \\\n-\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n-\t'\n-\t;;\n-linux-musl)\n-\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n-\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n-\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n-\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n-\t;;\n-linux-*|StaticAnalysis)\n-\t# Required so that apt doesn't wait for user input on certain packages.\n-\texport DEBIAN_FRONTEND=noninteractive\n-\n-\tapt update -q &&\n-\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n-\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n-\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n-\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n-\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n-\n-\tif test \"$jobname\" = StaticAnalysis\n-\tthen\n-\t\tapt install -q -y coccinelle\n-\tfi\n-\t;;\n-pedantic)\n-\tdnf -yq update >/dev/null &&\n-\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n-\t;;\n-esac\n-\n-end_group \"Install dependencies\"\n-- \n2.44.GIT\n\n"},{"id":"492499","messageId":"5240046a0fe604efa8840d023df4833c12bad952.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"[PATCH v2 08/12] ci: make Perforce binaries executable for all users","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:46:46Z","receivedAt":"2024-04-08T06:46:50Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The Perforce binaries are only made executable for the current user. On\nGitLab CI though we execute tests as a different user than \"root\", and\nthus these binaries may not be executable by that test user at all. This\nhas gone unnoticed so far because those binaries are optional -- in case\nthey don't exist we simply skip over tests requiring them.\n\nFix the setup so that we set the executable bits for all users.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex f4eb125fd2..068c478025 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -47,7 +47,7 @@ ubuntu-*)\n \tmkdir --parents \"$CUSTOM_PATH\"\n \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n \t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\tchmod u+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n+\tchmod a+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n \n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n-- \n2.44.GIT\n\n"},{"id":"492500","messageId":"29ceb623b9ec6a1d5b289c5d777f1e81adb4d0ad.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"[PATCH v2 09/12] ci: install JGit dependency","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:46:50Z","receivedAt":"2024-04-08T06:46:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have some tests in t5310 that use JGit to verify that bitmaps can be\nread both by Git and by JGit. We do not execute these tests in our CI\njobs though because we don't make JGit available there. Consequently,\nthe tests basically bitrot because almost nobody is ever going to have\nJGit in their path.\n\nInstall JGit to plug this test gap.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 15 ++++++++++++++-\n 1 file changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 068c478025..3fefe134ea 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -9,6 +9,7 @@ begin_group \"Install dependencies\"\n \n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n+JGITWHENCE=https://repo.eclipse.org/content/groups/releases//org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh\n \n # Make sudo a no-op and execute the command directly when running as root.\n # While using sudo would be fine on most platforms when we are root already,\n@@ -39,7 +40,7 @@ ubuntu-*)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install \\\n \t\tlanguage-pack-is libsvn-perl apache2 cvs cvsps git gnupg subversion \\\n-\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo default-jre \\\n \t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n \t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \\\n \t\t${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE\n@@ -52,6 +53,9 @@ ubuntu-*)\n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n+\n+\twget --quiet \"$JGITWHENCE\" --output-document=\"$CUSTOM_PATH/jgit\"\n+\tchmod a+x \"$CUSTOM_PATH/jgit\"\n \t;;\n ubuntu32-*)\n \tsudo linux32 --32bit i386 sh -c '\n@@ -112,6 +116,7 @@ then\n else\n \techo >&2 \"WARNING: perforce wasn't installed, see above for clues why\"\n fi\n+\n if type git-lfs >/dev/null 2>&1\n then\n \techo \"$(tput setaf 6)Git-LFS Version$(tput sgr0)\"\n@@ -120,4 +125,12 @@ else\n \techo >&2 \"WARNING: git-lfs wasn't installed, see above for clues why\"\n fi\n \n+if type jgit >/dev/null 2>&1\n+then\n+\techo \"$(tput setaf 6)JGit Version$(tput sgr0)\"\n+\tjgit version\n+else\n+\techo >&2 \"WARNING: JGit wasn't installed, see above for clues why\"\n+fi\n+\n end_group \"Install dependencies\"\n-- \n2.44.GIT\n\n"},{"id":"492501","messageId":"fc3472cdf300c3fc5d6270c3f2fc3cf0e9d3989f.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"[PATCH v2 10/12] t06xx: always execute backend-specific tests","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:46:55Z","receivedAt":"2024-04-08T06:46:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The tests in t06xx exercise specific ref formats. Next to probing some\nbasic functionality, these tests also exercise other low-level details\nspecific to the format. Those tests are only executed though in case\n`GIT_TEST_DEFAULT_REF_FORMAT` is set to the ref format of the respective\nbackend-under-test.\n\nIdeally, we would run the full test matrix for ref formats such that our\ncomplete test suite is executed with every supported format on every\nsupported platform. This is quite an expensive undertaking though, and\nthus we only execute e.g. the \"reftable\" tests on macOS and Linux. As a\nresult, we basically have no test coverage for the \"reftable\" format at\nall on other platforms like Windows.\n\nAdapt these tests so that they override `GIT_TEST_DEFAULT_REF_FORMAT`,\nwhich means that they'll always execute. This increases test coverage on\nplatforms that don't run the full test matrix, which at least gives us\nsome basic test coverage on those platforms for the \"reftable\" format.\n\nThis of course comes at the cost of running those tests multiple times\non platforms where we do run the full test matrix. But arguably, this is\na good thing because it will also cause us to e.g. run those tests with\nthe address sanitizer and other non-standard parameters.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0600-reffiles-backend.sh | 8 ++------\n t/t0601-reffiles-pack-refs.sh | 9 +++------\n t/t0610-reftable-basics.sh | 9 +++------\n 3 files changed, 8 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t0600-reffiles-backend.sh b/t/t0600-reffiles-backend.sh\nindex 64214340e7..a390cffc80 100755\n--- a/t/t0600-reffiles-backend.sh\n+++ b/t/t0600-reffiles-backend.sh\n@@ -4,16 +4,12 @@ test_description='Test reffiles backend'\n \n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n-if ! test_have_prereq REFFILES\n-then\n-\tskip_all='skipping reffiles specific tests'\n-\ttest_done\n-fi\n-\n test_expect_success 'setup' '\n \tgit commit --allow-empty -m Initial &&\n \tC=$(git rev-parse HEAD) &&\ndiff --git a/t/t0601-reffiles-pack-refs.sh b/t/t0601-reffiles-pack-refs.sh\nindex c309d2bae8..157f79fe52 100755\n--- a/t/t0601-reffiles-pack-refs.sh\n+++ b/t/t0601-reffiles-pack-refs.sh\n@@ -9,18 +9,15 @@ test_description='git pack-refs should not change the branch semantic\n This test runs git pack-refs and git show-ref and checks that the branch\n semantic is still the same.\n '\n+\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n-if ! test_have_prereq REFFILES\n-then\n-\tskip_all='skipping reffiles specific tests'\n-\ttest_done\n-fi\n-\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 686781192e..aa9282007c 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -4,17 +4,14 @@\n #\n \n test_description='reftable basics'\n+\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n . ./test-lib.sh\n \n-if ! test_have_prereq REFTABLE\n-then\n-\tskip_all='skipping reftable tests; set GIT_TEST_DEFAULT_REF_FORMAT=reftable'\n-\ttest_done\n-fi\n-\n INVALID_OID=$(test_oid 001)\n \n test_expect_success 'init: creates basic reftable structures' '\n-- \n2.44.GIT\n\n"},{"id":"492502","messageId":"cedf5929d1ea989cbf87bad48e35c0d191b624c3.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"[PATCH v2 11/12] t0610: fix non-portable variable assignment","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:47:00Z","receivedAt":"2024-04-08T06:47:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Older versions of the Dash shell fail to parse `local var=val`\nassignments in some cases when `val` is unquoted. Such failures can be\nobserved e.g. with Ubuntu 20.04 and older, which has a Dash version that\nstill has this bug.\n\nSuch an assignment has been introduced in t0610. The issue wasn't\ndetected for a while because this test used to only run when the\nGIT_TEST_DEFAULT_REF_FORMAT environment variable was set to \"refatble\".\nWe have dropped that requirement now though, meaning that it runs\nunconditionally, inclluding on jobs which use such older versions of\nUbuntu.\n\nWe have worked around such issues in the past, e.g. in ebee5580ca\n(parallel-checkout: avoid dash local bug in tests, 2021-06-06), by\nquoting the `val` side. Apply the same fix to t0610.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0610-reftable-basics.sh | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex aa9282007c..dfa7e274ea 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -78,9 +78,9 @@ test_expect_success 'init: reinitializing reftable with files backend fails' '\n '\n \n test_expect_perms () {\n-\tlocal perms=\"$1\"\n-\tlocal file=\"$2\"\n-\tlocal actual=$(ls -l \"$file\") &&\n+\tlocal perms=\"$1\" &&\n+\tlocal file=\"$2\" &&\n+\tlocal actual=\"$(ls -l \"$file\")\" &&\n \n \tcase \"$actual\" in\n \t$perms*)\n-- \n2.44.GIT\n\n"},{"id":"492503","messageId":"160b026e69547739a526fb6276a895904a4d33a8.1712555682.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"[PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T06:47:05Z","receivedAt":"2024-04-08T06:47:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While the reftable format is a recent introduction in Git, JGit already\nknows to read and write reftables since 2017. Given the complexity of\nthe format there is a very real risk of incompatibilities between those\ntwo implementations, which is something that we really want to avoid.\n\nAdd some basic tests that verify that reftables written by Git and JGit\ncan be read by the respective other implementation. For now this test\nsuite is rather small, only covering basic functionality. But it serves\nas a good starting point and can be extended over time.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0612-reftable-jgit-compatibility.sh | 132 +++++++++++++++++++++++++\n 1 file changed, 132 insertions(+)\n create mode 100755 t/t0612-reftable-jgit-compatibility.sh\n\ndiff --git a/t/t0612-reftable-jgit-compatibility.sh b/t/t0612-reftable-jgit-compatibility.sh\nnew file mode 100755\nindex 0000000000..222464e360\n--- /dev/null\n+++ b/t/t0612-reftable-jgit-compatibility.sh\n@@ -0,0 +1,132 @@\n+#!/bin/sh\n+\n+test_description='reftables are compatible with JGit'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+# JGit does not support the 'link' DIRC extension.\n+GIT_TEST_SPLIT_INDEX=0\n+export GIT_TEST_SPLIT_INDEX\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq JGIT\n+then\n+\tskip_all='skipping reftable JGit tests'\n+\ttest_done\n+fi\n+\n+if ! test_have_prereq SHA1\n+then\n+\tskip_all='skipping reftable JGit tests; JGit does not support SHA256 reftables'\n+\ttest_done\n+fi\n+\n+test_commit_jgit () {\n+\ttouch \"$1\" &&\n+\tjgit add \"$1\" &&\n+\tjgit commit -m \"$1\"\n+}\n+\n+test_same_refs () {\n+\tgit show-ref --head >cgit.actual &&\n+\tjgit show-ref >jgit-tabs.actual &&\n+\ttr \"\\t\" \" \" jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_same_ref () {\n+\tgit rev-parse \"$1\" >cgit.actual &&\n+\tjgit rev-parse \"$1\" >jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_same_reflog () {\n+\tgit reflog \"$*\" >cgit.actual &&\n+\tjgit reflog \"$*\" >jgit-newline.actual &&\n+\tsed '/^$/d' jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_expect_success 'CGit repository can be read by JGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\ttest_same_refs &&\n+\t\ttest_same_ref HEAD &&\n+\t\ttest_same_reflog HEAD\n+\t)\n+'\n+\n+test_expect_success 'JGit repository can be read by CGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tjgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttouch file &&\n+\t\tjgit add file &&\n+\t\tjgit commit -m \"initial commit\" &&\n+\n+\t\t# Note that we must convert the ref storage after we have\n+\t\t# written the default branch. Otherwise JGit will end up with\n+\t\t# no HEAD at all.\n+\t\tjgit convert-ref-storage --format=reftable &&\n+\n+\t\ttest_same_refs &&\n+\t\ttest_same_ref HEAD &&\n+\t\t# Interestingly, JGit cannot read its own reflog here. CGit can\n+\t\t# though.\n+\t\tprintf \"%s HEAD@{0}: commit (initial): initial commit\" \"$(git rev-parse --short HEAD)\" >expect &&\n+\t\tgit reflog HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'mixed writes from JGit and CGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit_jgit B &&\n+\t\ttest_commit C &&\n+\t\ttest_commit_jgit D &&\n+\n+\t\ttest_same_refs &&\n+\t\ttest_same_ref HEAD &&\n+\t\ttest_same_reflog HEAD\n+\t)\n+'\n+\n+test_expect_success 'JGit can read multi-level index' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\tawk \"\n+\t\t BEGIN {\n+\t\t\tprint \\\"start\\\";\n+\t\t\tfor (i = 0; i < 10000; i++)\n+\t\t\t printf \\\"create refs/heads/branch-%d HEAD\\n\\\", i;\n+\t\t\tprint \\\"commit\\\";\n+\t\t }\n+\t\t\" >input &&\n+\t\tgit update-ref --stdin \n---\n .github/workflows/main.yml | 2 +-\n ci/install-dependencies.sh | 2 +-\n ci/lib.sh | 4 ++--\n 3 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 3428773b09..684ef5c00d 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -303,7 +303,7 @@ jobs:\n CC: ${{matrix.vector.cc}}\n CC_PACKAGE: ${{matrix.vector.cc_package}}\n jobname: ${{matrix.vector.jobname}}\n- runs_on_pool: ${{matrix.vector.pool}}\n+ distro: ${{matrix.vector.pool}}\n runs-on: ${{matrix.vector.pool}}\n steps:\n - uses: actions/checkout@v4\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex b4e22de3cb..7d247b5ef4 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -11,7 +11,7 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n-case \"$runs_on_pool\" in\n+case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 0a73fc7bd1..d882250db5 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -279,7 +279,7 @@ then\n \n \tcache_dir=\"$HOME/none\"\n \n-\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n+\tdistro=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n \tJOBS=$(nproc)\n else\n \techo \"Could not identify CI type\" >&2\n@@ -318,7 +318,7 @@ export DEFAULT_TEST_TARGET=prove\n export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n-case \"$runs_on_pool\" in\n+case \"$distro\" in\n ubuntu-*)\n \tif test \"$jobname\" = \"linux-gcc-default\"\n \tthen\n-- \n2.44.GIT\n\n"},{"id":"492505","messageId":"CAPig+cR2cN=oKiy4T4_Ue85VEAWXwxq_3Rh3mHuYWNx2DFv_Lw@mail.gmail.com","threadId":"61266","inReplyTo":"cedf5929d1ea989cbf87bad48e35c0d191b624c3.1712555682.git.ps@pks.im","subject":"Re: [PATCH v2 11/12] t0610: fix non-portable variable assignment","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-04-08T06:57:04Z","receivedAt":"2024-04-08T06:57:16Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Apr 8, 2024 at 2:47 AM Patrick Steinhardt wrote:\n> Older versions of the Dash shell fail to parse `local var=val`\n> assignments in some cases when `val` is unquoted. Such failures can be\n> observed e.g. with Ubuntu 20.04 and older, which has a Dash version that\n> still has this bug.\n>\n> Such an assignment has been introduced in t0610. The issue wasn't\n> detected for a while because this test used to only run when the\n> GIT_TEST_DEFAULT_REF_FORMAT environment variable was set to \"refatble\".\n> We have dropped that requirement now though, meaning that it runs\n> unconditionally, inclluding on jobs which use such older versions of\n> Ubuntu.\n\ns/refatble/reftable/\ns/inclluding/including/\n\n> We have worked around such issues in the past, e.g. in ebee5580ca\n> (parallel-checkout: avoid dash local bug in tests, 2021-06-06), by\n> quoting the `val` side. Apply the same fix to t0610.\n>\n> Signed-off-by: Patrick Steinhardt \n"},{"id":"492506","messageId":"CAPig+cT3kvvHL+wLTFj58e5BnB7yBA=HD3C4vWC4zQhys3GCHA@mail.gmail.com","threadId":"61266","inReplyTo":"160b026e69547739a526fb6276a895904a4d33a8.1712555682.git.ps@pks.im","subject":"Re: [PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-04-08T07:10:59Z","receivedAt":"2024-04-08T07:11:11Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Apr 8, 2024 at 2:47 AM Patrick Steinhardt wrote:\n> While the reftable format is a recent introduction in Git, JGit already\n> knows to read and write reftables since 2017. Given the complexity of\n> the format there is a very real risk of incompatibilities between those\n> two implementations, which is something that we really want to avoid.\n>\n> Add some basic tests that verify that reftables written by Git and JGit\n> can be read by the respective other implementation. For now this test\n> suite is rather small, only covering basic functionality. But it serves\n> as a good starting point and can be extended over time.\n>\n> Signed-off-by: Patrick Steinhardt \n> ---\n> diff --git a/t/t0612-reftable-jgit-compatibility.sh b/t/t0612-reftable-jgit-compatibility.sh\n> @@ -0,0 +1,132 @@\n> +test_expect_success 'JGit can read multi-level index' '\n> + ...\n> + awk \"\n> + BEGIN {\n> + print \\\"start\\\";\n> + for (i = 0; i < 10000; i++)\n> + printf \\\"create refs/heads/branch-%d HEAD\\n\\\", i;\n> + print \\\"commit\\\";\n> + }\n> + \" >input &&\n\nI was going to suggest that you could accomplish this more easily\ndirectly in shell (without employing `awk`):\n\n {\n echo start &&\n printf \"create refs/heads/branch-%d HEAD\\n\" $(test_seq 0 9999) &&\n echo commit\n } >input &&\n\nbut then I realized that that could potentially run afoul of\ncommand-line length limit on some platform due to the 0-9999 sequence.\n"},{"id":"492508","messageId":"ZhOxTHSE3wBqKUmR@tanuki","threadId":"61266","inReplyTo":"CAPig+cR2cN=oKiy4T4_Ue85VEAWXwxq_3Rh3mHuYWNx2DFv_Lw@mail.gmail.com","subject":"Re: [PATCH v2 11/12] t0610: fix non-portable variable assignment","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T08:56:44Z","receivedAt":"2024-04-08T08:56:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 08, 2024 at 02:57:04AM -0400, Eric Sunshine wrote:\n> On Mon, Apr 8, 2024 at 2:47 AM Patrick Steinhardt wrote:\n> > Older versions of the Dash shell fail to parse `local var=val`\n> > assignments in some cases when `val` is unquoted. Such failures can be\n> > observed e.g. with Ubuntu 20.04 and older, which has a Dash version that\n> > still has this bug.\n> >\n> > Such an assignment has been introduced in t0610. The issue wasn't\n> > detected for a while because this test used to only run when the\n> > GIT_TEST_DEFAULT_REF_FORMAT environment variable was set to \"refatble\".\n> > We have dropped that requirement now though, meaning that it runs\n> > unconditionally, inclluding on jobs which use such older versions of\n> > Ubuntu.\n> \n> s/refatble/reftable/\n> s/inclluding/including/\n\nThanks, fixed locally. I'll wait for more feedback before sending out a\nnew version.\n\nPatrick\n\n> > We have worked around such issues in the past, e.g. in ebee5580ca\n> > (parallel-checkout: avoid dash local bug in tests, 2021-06-06), by\n> > quoting the `val` side. Apply the same fix to t0610.\n> >\n> > Signed-off-by: Patrick Steinhardt \n"},{"id":"492548","messageId":"xmqqpluzet2z.fsf@gitster.g","threadId":"61266","inReplyTo":"CAPig+cT3kvvHL+wLTFj58e5BnB7yBA=HD3C4vWC4zQhys3GCHA@mail.gmail.com","subject":"Re: [PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-08T16:07:32Z","receivedAt":"2024-04-08T16:07:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine writes:\n\n> I was going to suggest that you could accomplish this more easily\n> directly in shell (without employing `awk`):\n>\n> {\n> echo start &&\n> printf \"create refs/heads/branch-%d HEAD\\n\" $(test_seq 0 9999) &&\n> echo commit\n> } >input &&\n>\n> but then I realized that that could potentially run afoul of\n> command-line length limit on some platform due to the 0-9999 sequence.\n\nAs xargs is supposed to know the system limit, perhaps\n\n\ttest_seq 0 9999 | xargs printf \"...%d...\\n\"\n\nshould work?\n"},{"id":"492556","messageId":"CAPig+cQd7yfVVr+yZDxYY_oe8wC5OGt-SKtCnhCzZqJ0avP-Jg@mail.gmail.com","threadId":"61266","inReplyTo":"xmqqpluzet2z.fsf@gitster.g","subject":"Re: [PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-04-08T16:19:40Z","receivedAt":"2024-04-08T16:19:53Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Apr 8, 2024 at 12:07 PM Junio C Hamano wrote:\n> Eric Sunshine writes:\n> > I was going to suggest that you could accomplish this more easily\n> > directly in shell (without employing `awk`):\n> >\n> > {\n> > echo start &&\n> > printf \"create refs/heads/branch-%d HEAD\\n\" $(test_seq 0 9999) &&\n> > echo commit\n> > } >input &&\n> >\n> > but then I realized that that could potentially run afoul of\n> > command-line length limit on some platform due to the 0-9999 sequence.\n>\n> As xargs is supposed to know the system limit, perhaps\n>\n> test_seq 0 9999 | xargs printf \"...%d...\\n\"\n>\n> should work?\n\nHmm, yes, that should work nicely.\n\nWhether or not such a change is worthwhile is a different matter.\nAlthough it is perhaps simpler and easier to read, Windows folk might\nnot appreciate it since it spawns at least three processes (and\nperhaps more depending upon how test_seq is implemented), whereas the\n`awk` approach spawns a single process.\n"},{"id":"492558","messageId":"ZhQZsq5_NyXmWM-b@tanuki","threadId":"61266","inReplyTo":"xmqqpluzet2z.fsf@gitster.g","subject":"Re: [PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-08T16:22:10Z","receivedAt":"2024-04-08T16:22:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 08, 2024 at 09:07:32AM -0700, Junio C Hamano wrote:\n> Eric Sunshine writes:\n> \n> > I was going to suggest that you could accomplish this more easily\n> > directly in shell (without employing `awk`):\n> >\n> > {\n> > echo start &&\n> > printf \"create refs/heads/branch-%d HEAD\\n\" $(test_seq 0 9999) &&\n> > echo commit\n> > } >input &&\n> >\n> > but then I realized that that could potentially run afoul of\n> > command-line length limit on some platform due to the 0-9999 sequence.\n> \n> As xargs is supposed to know the system limit, perhaps\n> \n> \ttest_seq 0 9999 | xargs printf \"...%d...\\n\"\n> \n> should work?\n\nIs there a reason why we want to avoid using awk(1) in the first place?\nWe use it in several other tests and I don't think that the resulting\ncode is all that bad.\n\nPatrick\n"},{"id":"492559","messageId":"CAPig+cRERJGij+1486x3w97y6k=KHTSzp+FepbyUi080R6m1aw@mail.gmail.com","threadId":"61266","inReplyTo":"CAPig+cQd7yfVVr+yZDxYY_oe8wC5OGt-SKtCnhCzZqJ0avP-Jg@mail.gmail.com","subject":"Re: [PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2024-04-08T16:29:14Z","receivedAt":"2024-04-08T16:29:27Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Apr 8, 2024 at 12:19 PM Eric Sunshine wrote:\n> On Mon, Apr 8, 2024 at 12:07 PM Junio C Hamano wrote:\n> > Eric Sunshine writes:\n> > > I was going to suggest that you could accomplish this more easily\n> > > directly in shell (without employing `awk`):\n> > >\n> > > {\n> > > echo start &&\n> > > printf \"create refs/heads/branch-%d HEAD\\n\" $(test_seq 0 9999) &&\n> > > echo commit\n> > > } >input &&\n> > >\n> > > but then I realized that that could potentially run afoul of\n> > > command-line length limit on some platform due to the 0-9999 sequence.\n> >\n> > As xargs is supposed to know the system limit, perhaps\n> >\n> > test_seq 0 9999 | xargs printf \"...%d...\\n\"\n> >\n> > should work?\n>\n> Hmm, yes, that should work nicely.\n>\n> Whether or not such a change is worthwhile is a different matter.\n> Although it is perhaps simpler and easier to read, Windows folk might\n> not appreciate it since it spawns at least three processes (and\n> perhaps more depending upon how test_seq is implemented), whereas the\n> `awk` approach spawns a single process.\n\nAnd, to be clear, I have no objection to this patch's use of `awk`. It\nis \"good enough\" as is; I was not asking Patrick to make the change,\nbut rather made the comment in case the idea hadn't occurred to him.\n"},{"id":"492566","messageId":"20240408172638.GB1629595@coredump.intra.peff.net","threadId":"61266","inReplyTo":"ZhQZsq5_NyXmWM-b@tanuki","subject":"Re: [PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-04-08T17:26:38Z","receivedAt":"2024-04-08T17:26:39Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Apr 08, 2024 at 06:22:10PM +0200, Patrick Steinhardt wrote:\n\n> On Mon, Apr 08, 2024 at 09:07:32AM -0700, Junio C Hamano wrote:\n> > Eric Sunshine writes:\n> > \n> > > I was going to suggest that you could accomplish this more easily\n> > > directly in shell (without employing `awk`):\n> > >\n> > > {\n> > > echo start &&\n> > > printf \"create refs/heads/branch-%d HEAD\\n\" $(test_seq 0 9999) &&\n> > > echo commit\n> > > } >input &&\n> > >\n> > > but then I realized that that could potentially run afoul of\n> > > command-line length limit on some platform due to the 0-9999 sequence.\n> > \n> > As xargs is supposed to know the system limit, perhaps\n> > \n> > \ttest_seq 0 9999 | xargs printf \"...%d...\\n\"\n> > \n> > should work?\n> \n> Is there a reason why we want to avoid using awk(1) in the first place?\n> We use it in several other tests and I don't think that the resulting\n> code is all that bad.\n\nUsing awk is fine, but I think in general if we can do something just as\neasily without an extra process, that is preferable. Using xargs here\ndoes not to me count as \"just as easily\". However, using a shell loop\nmight actually be more readable because you'd avoid the extra quoting.\nI.e. either:\n\n for i in $(test_seq 10000)\n do\n\techo \"create refs/heads/branch-$i HEAD\"\n done\n\nor:\n\n i=0\n while test $i -lt 10000\n do\n\techo \"create refs/heads/branch-$i HEAD\"\n\ti=$((i+1))\n done\n\nI think the first one actually incurs an extra process anyway because of\nthe $(). The second one would not. Of course, the second one probably\nneeds &&-chaining and a \"|| return 1\" to work in our test snippet.\n\nIMHO the nicest thing would be if our test_seq could take a format\nparameter, like:\n\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 2eccf100c0..c8f32eb409 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1404,6 +1404,13 @@ test_cmp_fspath () {\n # from 1.\n \n test_seq () {\n+\tlocal fmt=\"%d\"\n+\tcase \"$1\" in\n+\t-f)\n+\t\tfmt=\"$2\"\n+\t\tshift 2\n+\t\t;;\n+\tesac\n \tcase $# in\n \t1)\tset 1 \"$@\" ;;\n \t2)\t;;\n@@ -1412,7 +1419,7 @@ test_seq () {\n \ttest_seq_counter__=$1\n \twhile test \"$test_seq_counter__\" -le \"$2\"\n \tdo\n-\t\techo \"$test_seq_counter__\"\n+\t\tprintf \"$fmt\\n\" \"$test_seq_counter__\"\n \t\ttest_seq_counter__=$(( $test_seq_counter__ + 1 ))\n \tdone\n }\n\nThen you could just write:\n\n test_seq -f \"create refs/heads/branch-%d HEAD\" 0 9999\n\nand I suspect there are several other spots which could be simplified as\nwell.\n\nAnyway, I don't think it is worth derailing your series for this, but\njust general food for thought.\n\n-Peff\n"},{"id":"492588","messageId":"xmqqfrvv8qvk.fsf@gitster.g","threadId":"61266","inReplyTo":"ZhQZsq5_NyXmWM-b@tanuki","subject":"Re: [PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-08T21:51:43Z","receivedAt":"2024-04-08T21:51:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> As xargs is supposed to know the system limit, perhaps\n>> \n>> \ttest_seq 0 9999 | xargs printf \"...%d...\\n\"\n>> \n>> should work?\n>\n> Is there a reason why we want to avoid using awk(1) in the first place?\n> We use it in several other tests and I don't think that the resulting\n> code is all that bad.\n\nThere is no reason. It was a canned reaction against \"gee we will\nbust shell's limit\" to \"use xargs then\".\n\nOf course, if we can do the loop in the shell and everything we do\nin the loop with shell builtins (printf is often builtin in modern\nshells but not always, if I recall correctly), that would be best,\nand if the job is _that_ simple that we could do in the shell, it\nwould make \"awk\" a horrible choice ;-)\n"},{"id":"492589","messageId":"xmqqbk6j8qth.fsf@gitster.g","threadId":"61266","inReplyTo":"20240408172638.GB1629595@coredump.intra.peff.net","subject":"Re: [PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-08T21:52:58Z","receivedAt":"2024-04-08T21:53:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I.e. either:\n>\n> for i in $(test_seq 10000)\n> do\n> \techo \"create refs/heads/branch-$i HEAD\"\n> done\n>\n> or:\n>\n> i=0\n> while test $i -lt 10000\n> do\n> \techo \"create refs/heads/branch-$i HEAD\"\n> \ti=$((i+1))\n> done\n>\n> I think the first one actually incurs an extra process anyway because of\n> the $(). The second one would not. Of course, the second one probably\n> needs &&-chaining and a \"|| return 1\" to work in our test snippet.\n> ...\n> Then you could just write:\n>\n> test_seq -f \"create refs/heads/branch-%d HEAD\" 0 9999\n>\n> and I suspect there are several other spots which could be simplified as\n> well.\n>\n> Anyway, I don't think it is worth derailing your series for this, but\n> just general food for thought.\n\nYup, I agree with everything you said.\n\nThanks.\n"},{"id":"492610","messageId":"xmqqle5n2sbf.fsf@gitster.g","threadId":"61266","inReplyTo":"cover.1712555682.git.ps@pks.im","subject":"Re: [PATCH v2 00/12] t: exercise Git/JGit reftable compatibility","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-09T02:17:08Z","receivedAt":"2024-04-09T02:17:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> CI runs for this series:\n>\n> - https://github.com/git/git/actions/runs/8595241646/\n> - https://gitlab.com/gitlab-org/git/-/pipelines/1243766428\n\nThanks.\n\nWhen this is queued on 'seen', I seem to be getting test errors from\nt0610. I suspect that it is not a breakage in this series, but the\nfact that this series enables reftable tests without setting any\nGIT_TEST_ environment variables, that causes an existing breakage\n(or two) more visible. I did not have time to locate which (other)\ntopic introduces the breakage, though.\n\n"},{"id":"492616","messageId":"ZhS5Ws7e0tjwlW6y@tanuki","threadId":"61266","inReplyTo":"xmqqle5n2sbf.fsf@gitster.g","subject":"Re: [PATCH v2 00/12] t: exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-09T03:43:22Z","receivedAt":"2024-04-09T03:43:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 08, 2024 at 07:17:08PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > CI runs for this series:\n> >\n> > - https://github.com/git/git/actions/runs/8595241646/\n> > - https://gitlab.com/gitlab-org/git/-/pipelines/1243766428\n> \n> Thanks.\n> \n> When this is queued on 'seen', I seem to be getting test errors from\n> t0610. I suspect that it is not a breakage in this series, but the\n> fact that this series enables reftable tests without setting any\n> GIT_TEST_ environment variables, that causes an existing breakage\n> (or two) more visible. I did not have time to locate which (other)\n> topic introduces the breakage, though.\n\nInteresting, I cannot reproduce any failures with the current state of\nseen (17ff004052 (Merge branch 'ps/ci-test-with-jgit' into seen,\n2024-04-08)). Was this on your local machine or in CI? Which platform\nare you using? Could you maybe provide logs of the failing tests?\n\nPatric\n"},{"id":"492620","messageId":"xmqqbk6j2j5u.fsf@gitster.g","threadId":"61266","inReplyTo":"ZhS5Ws7e0tjwlW6y@tanuki","subject":"Re: [PATCH v2 00/12] t: exercise Git/JGit reftable compatibility","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-09T05:34:53Z","receivedAt":"2024-04-09T05:34:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Mon, Apr 08, 2024 at 07:17:08PM -0700, Junio C Hamano wrote:\n>> Patrick Steinhardt writes:\n>> \n>> > CI runs for this series:\n>> >\n>> > - https://github.com/git/git/actions/runs/8595241646/\n>> > - https://gitlab.com/gitlab-org/git/-/pipelines/1243766428\n>> \n>> Thanks.\n>> \n>> When this is queued on 'seen', I seem to be getting test errors from\n>> t0610. I suspect that it is not a breakage in this series, but the\n>> fact that this series enables reftable tests without setting any\n>> GIT_TEST_ environment variables, that causes an existing breakage\n>> (or two) more visible. I did not have time to locate which (other)\n>> topic introduces the breakage, though.\n>\n> Interesting, I cannot reproduce any failures with the current state of\n> seen (17ff004052 (Merge branch 'ps/ci-test-with-jgit' into seen,\n> 2024-04-08)). Was this on your local machine or in CI? Which platform\n> are you using? Could you maybe provide logs of the failing tests?\n\nLocal execution on some variant of Debian testing.\n\nThe tests that fail are t0610.29 and t0610.30, both of which are\n\"honors core.sharedRepository\" with umask 002 and umask 022.\nThey expect \"-rw-rw-r--\", but actually get \"-rw-rw----\" when\nthe user's umask is 077 (or 007 for that matter).\n\nIf you add \"umask 002\" or \"umask 022\" near the beginning of the\nscript (e.g. before including test-lib.sh), the tests pass even when\nthe user's umask is 077 or 007 when the tests are run, of course, so\nthere is something the test must be doing in the test repository\nbefore it forces the umask to 002/022.\n\n"},{"id":"492623","messageId":"ZhTbBIdYrKdodL8M@tanuki","threadId":"61266","inReplyTo":"xmqqbk6j2j5u.fsf@gitster.g","subject":"Re: [PATCH v2 00/12] t: exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-09T06:07:00Z","receivedAt":"2024-04-09T06:07:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Apr 08, 2024 at 10:34:53PM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > On Mon, Apr 08, 2024 at 07:17:08PM -0700, Junio C Hamano wrote:\n> >> Patrick Steinhardt writes:\n> >> \n> >> > CI runs for this series:\n> >> >\n> >> > - https://github.com/git/git/actions/runs/8595241646/\n> >> > - https://gitlab.com/gitlab-org/git/-/pipelines/1243766428\n> >> \n> >> Thanks.\n> >> \n> >> When this is queued on 'seen', I seem to be getting test errors from\n> >> t0610. I suspect that it is not a breakage in this series, but the\n> >> fact that this series enables reftable tests without setting any\n> >> GIT_TEST_ environment variables, that causes an existing breakage\n> >> (or two) more visible. I did not have time to locate which (other)\n> >> topic introduces the breakage, though.\n> >\n> > Interesting, I cannot reproduce any failures with the current state of\n> > seen (17ff004052 (Merge branch 'ps/ci-test-with-jgit' into seen,\n> > 2024-04-08)). Was this on your local machine or in CI? Which platform\n> > are you using? Could you maybe provide logs of the failing tests?\n> \n> Local execution on some variant of Debian testing.\n> \n> The tests that fail are t0610.29 and t0610.30, both of which are\n> \"honors core.sharedRepository\" with umask 002 and umask 022.\n> They expect \"-rw-rw-r--\", but actually get \"-rw-rw----\" when\n> the user's umask is 077 (or 007 for that matter).\n> \n> If you add \"umask 002\" or \"umask 022\" near the beginning of the\n> script (e.g. before including test-lib.sh), the tests pass even when\n> the user's umask is 077 or 007 when the tests are run, of course, so\n> there is something the test must be doing in the test repository\n> before it forces the umask to 002/022.\n\nIndeed, I can reproduce the failures when I change my own umask. This\nissue is unrelated to this patch series -- as you mentioned, it is only\nbeing unmasked now because we increase test exposure by dropping the\nprereq.\n\nI'll send a separate patch series to fix this. Thanks!\n\nPatrick\n"},{"id":"492626","messageId":"cover.1712656576.git.ps@pks.im","threadId":"61266","inReplyTo":"xmqqle5n2sbf.fsf@gitster.g","subject":"[PATCH 0/2] t0610: fix umask tests","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-09T09:57:22Z","receivedAt":"2024-04-09T09:57:30Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthese patches address the issue reported by Junio where running t0610\nwith an umask of 0027 breaks some tests. The root cause of this is quite\nsimple: we executed git-pack-refs(1) outside of the shell that had the\nmodified umask, and thus we ran tests with the umask of the host shell\nby accident.\n\nThe series fixes this issue and extends the tests to exercise more\numasks than just 002 and 022, which would have caught the issue earlier.\n\nPatrick\n\nPatrick Steinhardt (2):\n t0610: make `--shared=` tests reusable\n t0610: execute git-pack-refs(1) with specified umask\n\n t/t0610-reftable-basics.sh | 67 ++++++++++++++++++++++----------------\n 1 file changed, 39 insertions(+), 28 deletions(-)\n\n-- \n2.44.GIT\n\n"},{"id":"492627","messageId":"f929e8ce00049d09052dbfc32869baaa50e68424.1712656576.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712656576.git.ps@pks.im","subject":"[PATCH 1/2] t0610: make `--shared=` tests reusable","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-09T09:57:27Z","receivedAt":"2024-04-09T09:57:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two kinds of `--shared=` tests, one for git-init(1) and one for\ngit-pack-refs(1). Merge them into a reusable function such that we can\neasily add additional testcases with different umasks and flags for the\n`--shared=` switch.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0610-reftable-basics.sh | 57 ++++++++++++++++++++------------------\n 1 file changed, 30 insertions(+), 27 deletions(-)\n\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 686781192e..a3269c20e0 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -96,23 +96,46 @@ test_expect_perms () {\n \tesac\n }\n \n-for umask in 002 022\n-do\n-\ttest_expect_success POSIXPERM 'init: honors core.sharedRepository' '\n+test_expect_reftable_perms () {\n+\tlocal umask=\"$1\"\n+\tlocal shared=\"$2\"\n+\tlocal expect=\"$3\"\n+\n+\ttest_expect_success POSIXPERM \"init: honors --shared=$shared with umask $umask\" '\n \t\ttest_when_finished \"rm -rf repo\" &&\n \t\t(\n \t\t\tumask $umask &&\n-\t\t\tgit init --shared=true repo &&\n+\t\t\tgit init --shared=$shared repo &&\n \t\t\ttest 1 = \"$(git -C repo config core.sharedrepository)\"\n \t\t) &&\n-\t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n+\t\ttest_expect_perms \"$expect\" repo/.git/reftable/tables.list &&\n \t\tfor table in repo/.git/reftable/*.ref\n \t\tdo\n-\t\t\ttest_expect_perms \"-rw-rw-r--\" \"$table\" ||\n+\t\t\ttest_expect_perms \"$expect\" \"$table\" ||\n \t\t\treturn 1\n \t\tdone\n \t'\n-done\n+\n+\ttest_expect_success POSIXPERM \"pack-refs: honors --shared=$shared with umask $umask\" '\n+\t\ttest_when_finished \"rm -rf repo\" &&\n+\t\t(\n+\t\t\tumask $umask &&\n+\t\t\tgit init --shared=$shared repo &&\n+\t\t\ttest_commit -C repo A &&\n+\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t) &&\n+\t\tgit -C repo pack-refs &&\n+\t\ttest_expect_perms \"$expect\" repo/.git/reftable/tables.list &&\n+\t\tfor table in repo/.git/reftable/*.ref\n+\t\tdo\n+\t\t\ttest_expect_perms \"$expect\" \"$table\" ||\n+\t\t\treturn 1\n+\t\tdone\n+\t'\n+}\n+\n+test_expect_reftable_perms 002 true \"-rw-rw-r--\"\n+test_expect_reftable_perms 022 true \"-rw-rw-r--\"\n \n test_expect_success 'clone: can clone reftable repository' '\n \ttest_when_finished \"rm -rf repo clone\" &&\n@@ -371,26 +394,6 @@ test_expect_success 'pack-refs: does not prune non-table files' '\n \ttest_path_is_file repo/.git/reftable/garbage\n '\n \n-for umask in 002 022\n-do\n-\ttest_expect_success POSIXPERM 'pack-refs: honors core.sharedRepository' '\n-\t\ttest_when_finished \"rm -rf repo\" &&\n-\t\t(\n-\t\t\tumask $umask &&\n-\t\t\tgit init --shared=true repo &&\n-\t\t\ttest_commit -C repo A &&\n-\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n-\t\t) &&\n-\t\tgit -C repo pack-refs &&\n-\t\ttest_expect_perms \"-rw-rw-r--\" repo/.git/reftable/tables.list &&\n-\t\tfor table in repo/.git/reftable/*.ref\n-\t\tdo\n-\t\t\ttest_expect_perms \"-rw-rw-r--\" \"$table\" ||\n-\t\t\treturn 1\n-\t\tdone\n-\t'\n-done\n-\n test_expect_success 'packed-refs: writes are synced' '\n \ttest_when_finished \"rm -rf repo\" &&\n \tgit init repo &&\n-- \n2.44.GIT\n\n"},{"id":"492628","messageId":"a3bfea94365138e85cac473a83eb51e821dd0def.1712656576.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712656576.git.ps@pks.im","subject":"[PATCH 2/2] t0610: execute git-pack-refs(1) with specified umask","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-09T09:57:32Z","receivedAt":"2024-04-09T09:57:38Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The tests for git-pack-refs(1) with the `core.sharedRepository` config\nexecute git-pack-refs(1) outside of the shell that has the expected\numask set. This is wrong because we want to test the behaviour of that\ncommand with different umasks. The issue went unnoticed because most\ndistributions have a default umask of 0022, and we only ever test with\n`--shared=true`, which re-adds the group write bit.\n\nFix the issue by moving git-pack-refs(1) into the umask'd shell and add\na bunch of test cases that exercise behaviour more thoroughly.\n\nNote that we drop the check for whether `core.sharedRepository` was set\nto the correct value to make the test setup a bit easier. We should be\nable to rely on git-init(1) doing its thing correctly. Furthermore, to\nhelp readability, we convert tests that pass `--shared=true` to instead\npass the equivalent `--shared=group`.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0610-reftable-basics.sh | 20 ++++++++++++++------\n 1 file changed, 14 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex a3269c20e0..8ffecefadc 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -105,8 +105,7 @@ test_expect_reftable_perms () {\n \t\ttest_when_finished \"rm -rf repo\" &&\n \t\t(\n \t\t\tumask $umask &&\n-\t\t\tgit init --shared=$shared repo &&\n-\t\t\ttest 1 = \"$(git -C repo config core.sharedrepository)\"\n+\t\t\tgit init --shared=$shared repo\n \t\t) &&\n \t\ttest_expect_perms \"$expect\" repo/.git/reftable/tables.list &&\n \t\tfor table in repo/.git/reftable/*.ref\n@@ -122,9 +121,9 @@ test_expect_reftable_perms () {\n \t\t\tumask $umask &&\n \t\t\tgit init --shared=$shared repo &&\n \t\t\ttest_commit -C repo A &&\n-\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list\n+\t\t\ttest_line_count = 3 repo/.git/reftable/tables.list &&\n+\t\t\tgit -C repo pack-refs\n \t\t) &&\n-\t\tgit -C repo pack-refs &&\n \t\ttest_expect_perms \"$expect\" repo/.git/reftable/tables.list &&\n \t\tfor table in repo/.git/reftable/*.ref\n \t\tdo\n@@ -134,8 +133,17 @@ test_expect_reftable_perms () {\n \t'\n }\n \n-test_expect_reftable_perms 002 true \"-rw-rw-r--\"\n-test_expect_reftable_perms 022 true \"-rw-rw-r--\"\n+test_expect_reftable_perms 002 umask \"-rw-rw-r--\"\n+test_expect_reftable_perms 022 umask \"-rw-r--r--\"\n+test_expect_reftable_perms 027 umask \"-rw-r-----\"\n+\n+test_expect_reftable_perms 002 group \"-rw-rw-r--\"\n+test_expect_reftable_perms 022 group \"-rw-rw-r--\"\n+test_expect_reftable_perms 027 group \"-rw-rw----\"\n+\n+test_expect_reftable_perms 002 world \"-rw-rw-r--\"\n+test_expect_reftable_perms 022 world \"-rw-rw-r--\"\n+test_expect_reftable_perms 027 world \"-rw-rw-r--\"\n \n test_expect_success 'clone: can clone reftable repository' '\n \ttest_when_finished \"rm -rf repo clone\" &&\n-- \n2.44.GIT\n\n"},{"id":"492726","messageId":"874jc9kvls.fsf@to1.studio","threadId":"61266","inReplyTo":"16603d40fdf96948580c04a7c2b791a97ec64fe7.1712555682.git.ps@pks.im","subject":"Re: [PATCH v2 03/12] ci: allow skipping sudo on dockerized jobs","fromName":"Toon claes","fromEmail":"toon@iotcl.com","sentAt":"2024-04-10T16:53:19Z","receivedAt":"2024-04-10T16:53:45Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"\n> [PATCH v2 03/12] ci: allow skipping sudo on dockerized jobs\n\nI think this title is somewhat misleading. While it's true, I don't\nthink it's limited to dockerized jobs. Something more along the lines of\n\"allow running install-dependencies.sh as root\" would make more sense to\nme.\n\nThat's the only tiny remark I have on this patch series.\n\n-- \nToon\n"},{"id":"492727","messageId":"xmqqjzl5uo9f.fsf@gitster.g","threadId":"61266","inReplyTo":"874jc9kvls.fsf@to1.studio","subject":"Re: [PATCH v2 03/12] ci: allow skipping sudo on dockerized jobs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-04-10T17:21:48Z","receivedAt":"2024-04-10T17:21:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Toon claes writes:\n\n>> [PATCH v2 03/12] ci: allow skipping sudo on dockerized jobs\n>\n> I think this title is somewhat misleading. While it's true, I don't\n> think it's limited to dockerized jobs. Something more along the lines of\n> \"allow running install-dependencies.sh as root\" would make more sense to\n> me.\n\nIt is true that dockerized is not the essential part of this change;\nand it is that we skip sudo when we are already root. So I agree\nwith you that the original title is misleading.\n\n\"allow running as root\" is somehow a bit unsatisfactory, though. It\nsounds as if _we_ were not allowing it before---what was preventing\nus from doing so was a system without sudo.\n\nStepping back a bit, I wonder if install-dependencies.sh will stay\nto be the only one among ci/ scripts that need to run things as\nroot. If we moved the new logic in this patch to ci/lib.sh that is\nincluded by everybody, then the title of the patch can become a\nshort and sweet\n\n ci: skip sudo when we are already root\n\n> That's the only tiny remark I have on this patch series.\n\nThanks.\n"},{"id":"492739","messageId":"t4powjtw4yapbivatafkez5e2ha5gcssintxam47ibkvjlb4le@ob5deeaxpzqr","threadId":"61266","inReplyTo":"160b026e69547739a526fb6276a895904a4d33a8.1712555682.git.ps@pks.im","subject":"Re: [PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-04-10T20:43:58Z","receivedAt":"2024-04-10T20:44:58Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/04/08 08:47AM, Patrick Steinhardt wrote:\n> While the reftable format is a recent introduction in Git, JGit already\n> knows to read and write reftables since 2017. Given the complexity of\n> the format there is a very real risk of incompatibilities between those\n> two implementations, which is something that we really want to avoid.\n> \n> Add some basic tests that verify that reftables written by Git and JGit\n> can be read by the respective other implementation. For now this test\n> suite is rather small, only covering basic functionality. But it serves\n> as a good starting point and can be extended over time.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> t/t0612-reftable-jgit-compatibility.sh | 132 +++++++++++++++++++++++++\n> 1 file changed, 132 insertions(+)\n> create mode 100755 t/t0612-reftable-jgit-compatibility.sh\n> \n> diff --git a/t/t0612-reftable-jgit-compatibility.sh b/t/t0612-reftable-jgit-compatibility.sh\n> new file mode 100755\n> index 0000000000..222464e360\n> --- /dev/null\n> +++ b/t/t0612-reftable-jgit-compatibility.sh\n> @@ -0,0 +1,132 @@\n> +#!/bin/sh\n> +\n> +test_description='reftables are compatible with JGit'\n> +\n> +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> +GIT_TEST_DEFAULT_REF_FORMAT=reftable\n> +export GIT_TEST_DEFAULT_REF_FORMAT\n> +\n> +# JGit does not support the 'link' DIRC extension.\n> +GIT_TEST_SPLIT_INDEX=0\n> +export GIT_TEST_SPLIT_INDEX\n> +\n> +. ./test-lib.sh\n> +\n> +if ! test_have_prereq JGIT\n\nDo we want these tests to run in CI? As far as I can tell these tests\nwould always be skipped.\n\n-Justin\n\n> +then\n> +\tskip_all='skipping reftable JGit tests'\n> +\ttest_done\n> +fi\n...\n"},{"id":"492740","messageId":"powdcoghaz226mefeeoinoyryzhooiqifrix76hocdd7scu42x@xbfkvqu5aa4b","threadId":"61266","inReplyTo":"6abc53bf5173e7de3fa271d175145c1672f17a88.1712555682.git.ps@pks.im","subject":"Re: [PATCH v2 05/12] ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-04-10T20:46:49Z","receivedAt":"2024-04-10T20:47:50Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/04/08 08:46AM, Patrick Steinhardt wrote:\n> We're about to merge the \"install-docker-dependencies.sh\" script into\n> \"install-dependencies.sh\". This will also move our Alpine-based jobs\n> over to use the latter script. This script uses the Bash shell though,\n> which is not available by default on Alpine Linux.\n> \n> Refactor \"install-dependencies.sh\" to use \"/bin/sh\" instead of Bash.\n> This requires us to get rid of the pushd/popd invocations, which are\n> replaced by some more elaborate commands that download or extract\n> executables right to where they are needed.\n> \n> Signed-off-by: Patrick Steinhardt \n> ---\n> ci/install-dependencies.sh | 28 +++++++++++-----------------\n> 1 file changed, 11 insertions(+), 17 deletions(-)\n> \n> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> index fad53aac96..7bcccc96fd 100755\n> --- a/ci/install-dependencies.sh\n> +++ b/ci/install-dependencies.sh\n> @@ -1,4 +1,4 @@\n> -#!/usr/bin/env bash\n> +#!/bin/sh\n> #\n> # Install dependencies required to build and test Git on Linux and macOS\n> #\n> @@ -30,19 +30,14 @@ ubuntu-*)\n> \t\t$CC_PACKAGE $PYTHON_PACKAGE\n> \n> \tmkdir --parents \"$P4_PATH\"\n> -\tpushd \"$P4_PATH\"\n> -\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n> -\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4\"\n> -\t\tchmod u+x p4d\n> -\t\tchmod u+x p4\n> -\tpopd\n> +\twget --quiet --directory-prefix=\"$P4_PATH\" \\\n> +\t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n> +\tchmod u+x \"$P4_PATH/p4d\" \"$P4_PATH/p4\"\n> \n> \tmkdir --parents \"$GIT_LFS_PATH\"\n> -\tpushd \"$GIT_LFS_PATH\"\n> -\t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n> -\t\ttar --extract --gunzip --file \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n> -\t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n> -\tpopd\n> +\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n> +\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n\nDo we want to fold this line since it is rather long?\n\n-Justin\n\n"},{"id":"492747","messageId":"zmr3qstsa3egnt7qi3pvaonltk4aoqr3x3vzch7esqjj7vuyj7@vbxldlnl6h4k","threadId":"61266","inReplyTo":"cover.1712656576.git.ps@pks.im","subject":"Re: [PATCH 0/2] t0610: fix umask tests","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-04-10T21:49:21Z","receivedAt":"2024-04-10T21:50:22Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/04/09 11:57AM, Patrick Steinhardt wrote:\n> Hi,\n> \n> these patches address the issue reported by Junio where running t0610\n> with an umask of 0027 breaks some tests. The root cause of this is quite\n> simple: we executed git-pack-refs(1) outside of the shell that had the\n> modified umask, and thus we ran tests with the umask of the host shell\n> by accident.\n> \n> The series fixes this issue and extends the tests to exercise more\n> umasks than just 002 and 022, which would have caught the issue earlier.\n> \n> Patrick\n> \n> Patrick Steinhardt (2):\n> t0610: make `--shared=` tests reusable\n> t0610: execute git-pack-refs(1) with specified umask\n> \n> t/t0610-reftable-basics.sh | 67 ++++++++++++++++++++++----------------\n> 1 file changed, 39 insertions(+), 28 deletions(-)\n> \n> -- \n> 2.44.GIT\n> \n\nNice fix! I like how the tests were made more extensible too. This looks\ngood to me :)\n\n-Justin\n\n"},{"id":"492760","messageId":"ZhelcuEL_D8RRtQb@tanuki","threadId":"61266","inReplyTo":"powdcoghaz226mefeeoinoyryzhooiqifrix76hocdd7scu42x@xbfkvqu5aa4b","subject":"Re: [PATCH v2 05/12] ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T08:55:14Z","receivedAt":"2024-04-11T08:55:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 10, 2024 at 03:46:49PM -0500, Justin Tobler wrote:\n> On 24/04/08 08:46AM, Patrick Steinhardt wrote:\n> > We're about to merge the \"install-docker-dependencies.sh\" script into\n> > \"install-dependencies.sh\". This will also move our Alpine-based jobs\n> > over to use the latter script. This script uses the Bash shell though,\n> > which is not available by default on Alpine Linux.\n> > \n> > Refactor \"install-dependencies.sh\" to use \"/bin/sh\" instead of Bash.\n> > This requires us to get rid of the pushd/popd invocations, which are\n> > replaced by some more elaborate commands that download or extract\n> > executables right to where they are needed.\n> > \n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > ci/install-dependencies.sh | 28 +++++++++++-----------------\n> > 1 file changed, 11 insertions(+), 17 deletions(-)\n> > \n> > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> > index fad53aac96..7bcccc96fd 100755\n> > --- a/ci/install-dependencies.sh\n> > +++ b/ci/install-dependencies.sh\n> > @@ -1,4 +1,4 @@\n> > -#!/usr/bin/env bash\n> > +#!/bin/sh\n> > #\n> > # Install dependencies required to build and test Git on Linux and macOS\n> > #\n> > @@ -30,19 +30,14 @@ ubuntu-*)\n> > \t\t$CC_PACKAGE $PYTHON_PACKAGE\n> > \n> > \tmkdir --parents \"$P4_PATH\"\n> > -\tpushd \"$P4_PATH\"\n> > -\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n> > -\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4\"\n> > -\t\tchmod u+x p4d\n> > -\t\tchmod u+x p4\n> > -\tpopd\n> > +\twget --quiet --directory-prefix=\"$P4_PATH\" \\\n> > +\t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n> > +\tchmod u+x \"$P4_PATH/p4d\" \"$P4_PATH/p4\"\n> > \n> > \tmkdir --parents \"$GIT_LFS_PATH\"\n> > -\tpushd \"$GIT_LFS_PATH\"\n> > -\t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n> > -\t\ttar --extract --gunzip --file \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n> > -\t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n> > -\tpopd\n> > +\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n> > +\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n> \n> Do we want to fold this line since it is rather long?\n> \n> -Justin\n\nYeah, let's.\n\nPatrick\n"},{"id":"492761","messageId":"ZhelfC2wJmcnWr7L@tanuki","threadId":"61266","inReplyTo":"xmqqjzl5uo9f.fsf@gitster.g","subject":"Re: [PATCH v2 03/12] ci: allow skipping sudo on dockerized jobs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T08:55:24Z","receivedAt":"2024-04-11T08:55:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 10, 2024 at 10:21:48AM -0700, Junio C Hamano wrote:\n> Toon claes writes:\n> \n> >> [PATCH v2 03/12] ci: allow skipping sudo on dockerized jobs\n> >\n> > I think this title is somewhat misleading. While it's true, I don't\n> > think it's limited to dockerized jobs. Something more along the lines of\n> > \"allow running install-dependencies.sh as root\" would make more sense to\n> > me.\n> \n> It is true that dockerized is not the essential part of this change;\n> and it is that we skip sudo when we are already root. So I agree\n> with you that the original title is misleading.\n> \n> \"allow running as root\" is somehow a bit unsatisfactory, though. It\n> sounds as if _we_ were not allowing it before---what was preventing\n> us from doing so was a system without sudo.\n> \n> Stepping back a bit, I wonder if install-dependencies.sh will stay\n> to be the only one among ci/ scripts that need to run things as\n> root. If we moved the new logic in this patch to ci/lib.sh that is\n> included by everybody, then the title of the patch can become a\n> short and sweet\n> \n> ci: skip sudo when we are already root\n\nAgreed, this title reads much better than what I had.\n\n> > That's the only tiny remark I have on this patch series.\n> \n> Thanks.\n\nThanks to both of you!\n\nPatrick\n"},{"id":"492762","messageId":"ZhelldbAia0wxIpD@tanuki","threadId":"61266","inReplyTo":"t4powjtw4yapbivatafkez5e2ha5gcssintxam47ibkvjlb4le@ob5deeaxpzqr","subject":"Re: [PATCH v2 12/12] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T08:55:49Z","receivedAt":"2024-04-11T08:55:54Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Apr 10, 2024 at 03:43:58PM -0500, Justin Tobler wrote:\n> On 24/04/08 08:47AM, Patrick Steinhardt wrote:\n> > While the reftable format is a recent introduction in Git, JGit already\n> > knows to read and write reftables since 2017. Given the complexity of\n> > the format there is a very real risk of incompatibilities between those\n> > two implementations, which is something that we really want to avoid.\n> > \n> > Add some basic tests that verify that reftables written by Git and JGit\n> > can be read by the respective other implementation. For now this test\n> > suite is rather small, only covering basic functionality. But it serves\n> > as a good starting point and can be extended over time.\n> > \n> > Signed-off-by: Patrick Steinhardt \n> > ---\n> > t/t0612-reftable-jgit-compatibility.sh | 132 +++++++++++++++++++++++++\n> > 1 file changed, 132 insertions(+)\n> > create mode 100755 t/t0612-reftable-jgit-compatibility.sh\n> > \n> > diff --git a/t/t0612-reftable-jgit-compatibility.sh b/t/t0612-reftable-jgit-compatibility.sh\n> > new file mode 100755\n> > index 0000000000..222464e360\n> > --- /dev/null\n> > +++ b/t/t0612-reftable-jgit-compatibility.sh\n> > @@ -0,0 +1,132 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='reftables are compatible with JGit'\n> > +\n> > +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> > +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> > +GIT_TEST_DEFAULT_REF_FORMAT=reftable\n> > +export GIT_TEST_DEFAULT_REF_FORMAT\n> > +\n> > +# JGit does not support the 'link' DIRC extension.\n> > +GIT_TEST_SPLIT_INDEX=0\n> > +export GIT_TEST_SPLIT_INDEX\n> > +\n> > +. ./test-lib.sh\n> > +\n> > +if ! test_have_prereq JGIT\n> \n> Do we want these tests to run in CI? As far as I can tell these tests\n> would always be skipped.\n\nThey aren't skipped on Linux-based jobs at least. The JGIT prerequisite\nis defined in \"t/test-lib.sh\" like so:\n\n```\ntest_lazy_prereq JGIT '\n\tjgit --version\n'\n```\n\nAnd because we have adapted \"install-dependencies.sh\" to install the\n\"jgit\" executable into our custom PATH directory it is available,\nmeaning that the prereq is fulfilled.\n\nBut you're actually onto something: while the tests run on GitHub, they\ndon't run on GitLab. And this is caused by the unprivileged build that\nwe use on GitLab, where we install dependencies as a different user than\nwhat we run tests with. Consequently, as the custom path is derived from\nHOME, and HOME changes, we cannot find these binaries.\n\nThis is not a new issue, it's been there since the inception of the\nGitLab pipelines. I'll include another patch on top to fix this.\n\nPatrick\n"},{"id":"492763","messageId":"cover.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH v3 00/13] t: exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:09:52Z","receivedAt":"2024-04-11T09:09:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the third version of my patch series that introduces\ncompatibility checks for the CGit/JGit refatble implementations.\n\nChanges compared to v2:\n\n - Adjusted commit subject for the skipping-sudo patch.\n\n - Broken an overly long line in \"install-dependencies.sh\".\n\n - Fixed tests that depend on custom binaries not running on GitLab\n CI due to the unprivileged-user setup.\n\n - Fixed some typos.\n\nCI jobs for this version:\n\n - https://gitlab.com/gitlab-org/git/-/pipelines/1248875682\n - https://github.com/git/git/actions/runs/8643984413\n\nThanks for all the feedback!\n\nPatrick\n\nPatrick Steinhardt (13):\n ci: rename \"runs_on_pool\" to \"distro\"\n ci: expose distro name in dockerized GitHub jobs\n ci: skip sudo when we are already root\n ci: drop duplicate package installation for \"linux-gcc-default\"\n ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"\n ci: merge custom PATH directories\n ci: fix setup of custom path for GitLab CI\n ci: merge scripts which install dependencies\n ci: make Perforce binaries executable for all users\n ci: install JGit dependency\n t06xx: always execute backend-specific tests\n t0610: fix non-portable variable assignment\n t0612: add tests to exercise Git/JGit reftable compatibility\n\n .github/workflows/main.yml | 8 +-\n .gitlab-ci.yml | 6 +-\n ci/install-dependencies.sh | 101 +++++++++++++------\n ci/install-docker-dependencies.sh | 46 ---------\n ci/lib.sh | 14 +--\n t/t0600-reffiles-backend.sh | 8 +-\n t/t0601-reffiles-pack-refs.sh | 9 +-\n t/t0610-reftable-basics.sh | 15 ++-\n t/t0612-reftable-jgit-compatibility.sh | 132 +++++++++++++++++++++++++\n 9 files changed, 229 insertions(+), 110 deletions(-)\n delete mode 100755 ci/install-docker-dependencies.sh\n create mode 100755 t/t0612-reftable-jgit-compatibility.sh\n\nRange-diff against v2:\n 1: 89723b6812 = 1: 46502bbe22 ci: rename \"runs_on_pool\" to \"distro\"\n 2: e60a40bd65 = 2: d076ed9857 ci: expose distro name in dockerized GitHub jobs\n 3: 16603d40fd ! 3: cc0c29052f ci: allow skipping sudo on dockerized jobs\n @@ Metadata\n Author: Patrick Steinhardt \n \n ## Commit message ##\n - ci: allow skipping sudo on dockerized jobs\n + ci: skip sudo when we are already root\n \n Our \"install-dependencies.sh\" script is executed by non-dockerized jobs\n to install dependencies. These jobs don't run with \"root\" permissions,\n 4: b4f6d6d3bf = 4: 803f5020e0 ci: drop duplicate package installation for \"linux-gcc-default\"\n 5: 6abc53bf51 ! 5: d2745e9438 ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"\n @@ ci/install-dependencies.sh: ubuntu-*)\n -\t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n -\tpopd\n +\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n -+\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n ++\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n ++\t\t-C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n +\trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n macos-*)\n 6: d9be4db56f ! 6: 61f108d954 ci: merge custom PATH directories\n @@ ci/install-dependencies.sh: ubuntu-*)\n \n -\tmkdir --parents \"$GIT_LFS_PATH\"\n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n --\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n -+\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n + \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n +-\t\t-C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n ++\t\t-C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n macos-*)\n -: ---------- > 7: ef61b578da ci: fix setup of custom path for GitLab CI\n 7: 4a90c003d1 ! 8: 7748f87f8c ci: merge scripts which install dependencies\n @@ .github/workflows/main.yml: jobs:\n if: failure() && env.FAILED_TEST_ARTIFACTS != ''\n \n ## .gitlab-ci.yml ##\n -@@ .gitlab-ci.yml: workflow:\n - test:linux:\n - image: $image\n +@@ .gitlab-ci.yml: test:linux:\n + variables:\n + CUSTOM_PATH: \"/custom\"\n before_script:\n - - ./ci/install-docker-dependencies.sh\n + - ./ci/install-dependencies.sh\n @@ ci/install-dependencies.sh: then\n \tmkdir --parents \"$CUSTOM_PATH\"\n \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n @@ ci/install-dependencies.sh: ubuntu-*)\n - \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n + \t\t-C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n +ubuntu32-*)\n 8: 5240046a0f ! 9: f7399382f2 ci: make Perforce binaries executable for all users\n @@ ci/install-dependencies.sh: ubuntu-*)\n +\tchmod a+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n \n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n - \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n + \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n 9: 29ceb623b9 ! 10: b835ff8b78 ci: install JGit dependency\n @@ ci/install-dependencies.sh: ubuntu-*)\n \t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \\\n \t\t${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE\n @@ ci/install-dependencies.sh: ubuntu-*)\n - \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n - \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" -C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n + \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n + \t\t-C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n +\n +\twget --quiet \"$JGITWHENCE\" --output-document=\"$CUSTOM_PATH/jgit\"\n10: fc3472cdf3 = 11: 7136c8b6c2 t06xx: always execute backend-specific tests\n11: cedf5929d1 ! 12: cf4ee9c427 t0610: fix non-portable variable assignment\n @@ Commit message\n \n Such an assignment has been introduced in t0610. The issue wasn't\n detected for a while because this test used to only run when the\n - GIT_TEST_DEFAULT_REF_FORMAT environment variable was set to \"refatble\".\n + GIT_TEST_DEFAULT_REF_FORMAT environment variable was set to \"reftable\".\n We have dropped that requirement now though, meaning that it runs\n - unconditionally, inclluding on jobs which use such older versions of\n + unconditionally, including on jobs which use such older versions of\n Ubuntu.\n \n We have worked around such issues in the past, e.g. in ebee5580ca\n12: 160b026e69 ! 13: a9cd20eebc t0612: add tests to exercise Git/JGit reftable compatibility\n @@ t/t0612-reftable-jgit-compatibility.sh (new)\n +\n +if ! test_have_prereq JGIT\n +then\n -+\tskip_all='skipping reftable JGit tests'\n ++\tskip_all='skipping reftable JGit tests; JGit is not present in PATH'\n +\ttest_done\n +fi\n +\n-- \n2.44.GIT\n\n"},{"id":"492764","messageId":"46502bbe22e3404d41ab8da152d9498c607470b7.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 01/13] ci: rename \"runs_on_pool\" to \"distro\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:09:57Z","receivedAt":"2024-04-11T09:10:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"runs_on_pool\" environment variable is used by our CI scripts to\ndistinguish the different kinds of operating systems. It is quite\nspecific to GitHub Actions though and not really a descriptive name.\n\nRename the variable to \"distro\" to clarify its intent.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 2 +-\n ci/install-dependencies.sh | 2 +-\n ci/lib.sh | 4 ++--\n 3 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 3428773b09..684ef5c00d 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -303,7 +303,7 @@ jobs:\n CC: ${{matrix.vector.cc}}\n CC_PACKAGE: ${{matrix.vector.cc_package}}\n jobname: ${{matrix.vector.jobname}}\n- runs_on_pool: ${{matrix.vector.pool}}\n+ distro: ${{matrix.vector.pool}}\n runs-on: ${{matrix.vector.pool}}\n steps:\n - uses: actions/checkout@v4\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex b4e22de3cb..7d247b5ef4 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -11,7 +11,7 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n-case \"$runs_on_pool\" in\n+case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 0a73fc7bd1..d882250db5 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -279,7 +279,7 @@ then\n \n \tcache_dir=\"$HOME/none\"\n \n-\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n+\tdistro=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n \tJOBS=$(nproc)\n else\n \techo \"Could not identify CI type\" >&2\n@@ -318,7 +318,7 @@ export DEFAULT_TEST_TARGET=prove\n export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n-case \"$runs_on_pool\" in\n+case \"$distro\" in\n ubuntu-*)\n \tif test \"$jobname\" = \"linux-gcc-default\"\n \tthen\n-- \n2.44.GIT\n\n"},{"id":"492765","messageId":"d076ed9857635bcb29d3ac0717d3b5a3cc8f425c.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 02/13] ci: expose distro name in dockerized GitHub jobs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:10:03Z","receivedAt":"2024-04-11T09:10:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Expose a distro name in dockerized jobs. This will be used in a\nsubsequent commit where we merge the installation scripts for dockerized\nand non-dockerized jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 684ef5c00d..71cd4e5486 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -342,12 +342,16 @@ jobs:\n vector:\n - jobname: linux-musl\n image: alpine\n+ distro: alpine-latest\n - jobname: linux32\n image: daald/ubuntu32:xenial\n+ distro: ubuntu32-16.04\n - jobname: pedantic\n image: fedora\n+ distro: fedora-latest\n env:\n jobname: ${{matrix.vector.jobname}}\n+ distro: ${{matrix.vector.distro}}\n runs-on: ubuntu-latest\n container: ${{matrix.vector.image}}\n steps:\n-- \n2.44.GIT\n\n"},{"id":"492766","messageId":"cc0c29052f432bdb046b3193ffb04bb8c2f83f37.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 03/13] ci: skip sudo when we are already root","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:10:07Z","receivedAt":"2024-04-11T09:10:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our \"install-dependencies.sh\" script is executed by non-dockerized jobs\nto install dependencies. These jobs don't run with \"root\" permissions,\nbut with a separate user. Consequently, we need to use sudo(8) there to\nelevate permissions when installing packages.\n\nWe're about to merge \"install-docker-dependencies.sh\" into that script\nthough, and our Docker containers do run as \"root\". Using sudo(8) is\nthus unnecessary there, even though it would be harmless. On some images\nlike Alpine Linux though there is no sudo(8) available by default, which\nwould consequently break the build.\n\nAdapt the script to make \"sudo\" a no-op when running as \"root\" user.\nThis allows us to easily reuse the script for our dockerized jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7d247b5ef4..7dfd3e50ed 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -11,6 +11,17 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n+# Make sudo a no-op and execute the command directly when running as root.\n+# While using sudo would be fine on most platforms when we are root already,\n+# some platforms like e.g. Alpine Linux do not have sudo available by default\n+# and would thus break.\n+if test \"$(id -u)\" -eq 0\n+then\n+\tsudo () {\n+\t\t\"$@\"\n+\t}\n+fi\n+\n case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n-- \n2.44.GIT\n\n"},{"id":"492767","messageId":"803f5020e04da1ea84691a63e97afbc8b99bd3d4.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 04/13] ci: drop duplicate package installation for \"linux-gcc-default\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:10:13Z","receivedAt":"2024-04-11T09:10:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"linux-gcc-default\" job installs common Ubuntu packages. This is\nalready done in the distro-specific switch, so we basically duplicate\nthe effort here.\n\nDrop the duplicate package installations and inline the variable that\ncontains those common packages.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 17 ++++++++---------\n 1 file changed, 8 insertions(+), 9 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7dfd3e50ed..fad53aac96 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -7,9 +7,6 @@\n \n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n-UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n- tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n- libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n # Make sudo a no-op and execute the command directly when running as root.\n # While using sudo would be fine on most platforms when we are root already,\n@@ -25,8 +22,13 @@ fi\n case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n-\tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n-\t\t$UBUNTU_COMMON_PKGS $CC_PACKAGE $PYTHON_PACKAGE\n+\tsudo apt-get -q -y install \\\n+\t\tlanguage-pack-is libsvn-perl apache2 \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev \\\n+\t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n+\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n+\t\t$CC_PACKAGE $PYTHON_PACKAGE\n+\n \tmkdir --parents \"$P4_PATH\"\n \tpushd \"$P4_PATH\"\n \t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n@@ -34,6 +36,7 @@ ubuntu-*)\n \t\tchmod u+x p4d\n \t\tchmod u+x p4\n \tpopd\n+\n \tmkdir --parents \"$GIT_LFS_PATH\"\n \tpushd \"$GIT_LFS_PATH\"\n \t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n@@ -83,10 +86,6 @@ Documentation)\n \ttest -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n \tsudo gem install --version 1.5.8 asciidoctor\n \t;;\n-linux-gcc-default)\n-\tsudo apt-get -q update\n-\tsudo apt-get -q -y install $UBUNTU_COMMON_PKGS\n-\t;;\n esac\n \n if type p4d >/dev/null 2>&1 && type p4 >/dev/null 2>&1\n-- \n2.44.GIT\n\n"},{"id":"492768","messageId":"d2745e9438199217c1097406102a84ed021e4d2a.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 05/13] ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:10:18Z","receivedAt":"2024-04-11T09:10:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to merge the \"install-docker-dependencies.sh\" script into\n\"install-dependencies.sh\". This will also move our Alpine-based jobs\nover to use the latter script. This script uses the Bash shell though,\nwhich is not available by default on Alpine Linux.\n\nRefactor \"install-dependencies.sh\" to use \"/bin/sh\" instead of Bash.\nThis requires us to get rid of the pushd/popd invocations, which are\nreplaced by some more elaborate commands that download or extract\nexecutables right to where they are needed.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 29 ++++++++++++-----------------\n 1 file changed, 12 insertions(+), 17 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex fad53aac96..2d6af876d6 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -1,4 +1,4 @@\n-#!/usr/bin/env bash\n+#!/bin/sh\n #\n # Install dependencies required to build and test Git on Linux and macOS\n #\n@@ -30,19 +30,15 @@ ubuntu-*)\n \t\t$CC_PACKAGE $PYTHON_PACKAGE\n \n \tmkdir --parents \"$P4_PATH\"\n-\tpushd \"$P4_PATH\"\n-\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n-\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\t\tchmod u+x p4d\n-\t\tchmod u+x p4\n-\tpopd\n+\twget --quiet --directory-prefix=\"$P4_PATH\" \\\n+\t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n+\tchmod u+x \"$P4_PATH/p4d\" \"$P4_PATH/p4\"\n \n \tmkdir --parents \"$GIT_LFS_PATH\"\n-\tpushd \"$GIT_LFS_PATH\"\n-\t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\t\ttar --extract --gunzip --file \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n-\tpopd\n+\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n+\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n+\t\t-C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n+\trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n macos-*)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n@@ -53,11 +49,10 @@ macos-*)\n \tbrew link --force gettext\n \n \tmkdir -p \"$P4_PATH\"\n-\tpushd \"$P4_PATH\"\n-\t\twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n-\t\ttar -xf helix-core-server.tgz &&\n-\t\tsudo xattr -d com.apple.quarantine p4 p4d 2>/dev/null || true\n-\tpopd\n+\twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n+\ttar -xf helix-core-server.tgz -C \"$P4_PATH\" p4 p4d &&\n+\tsudo xattr -d com.apple.quarantine \"$P4_PATH/p4\" \"$P4_PATH/p4d\" 2>/dev/null || true\n+\trm helix-core-server.tgz\n \n \tif test -n \"$CC_PACKAGE\"\n \tthen\n-- \n2.44.GIT\n\n"},{"id":"492769","messageId":"61f108d95463f94b3921669a276472bd01fdc606.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 06/13] ci: merge custom PATH directories","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:10:26Z","receivedAt":"2024-04-11T09:10:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're downloading various executables required by our tests. Each of\nthese executables goes into its own directory, which is then appended to\nthe PATH variable. Consequently, whenever we add a new dependency and\nthus a new directory, we would have to adapt to this change in several\nplaces.\n\nRefactor this to instead put all binaries into a single directory.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 15 +++++++--------\n ci/lib.sh | 10 +++-------\n 2 files changed, 10 insertions(+), 15 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 2d6af876d6..bafe37f2d1 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -29,15 +29,14 @@ ubuntu-*)\n \t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n \t\t$CC_PACKAGE $PYTHON_PACKAGE\n \n-\tmkdir --parents \"$P4_PATH\"\n-\twget --quiet --directory-prefix=\"$P4_PATH\" \\\n+\tmkdir --parents \"$CUSTOM_PATH\"\n+\twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n \t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\tchmod u+x \"$P4_PATH/p4d\" \"$P4_PATH/p4\"\n+\tchmod u+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n \n-\tmkdir --parents \"$GIT_LFS_PATH\"\n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n-\t\t-C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n+\t\t-C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n macos-*)\n@@ -48,10 +47,10 @@ macos-*)\n \tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \n-\tmkdir -p \"$P4_PATH\"\n+\tmkdir -p \"$CUSTOM_PATH\"\n \twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n-\ttar -xf helix-core-server.tgz -C \"$P4_PATH\" p4 p4d &&\n-\tsudo xattr -d com.apple.quarantine \"$P4_PATH/p4\" \"$P4_PATH/p4d\" 2>/dev/null || true\n+\ttar -xf helix-core-server.tgz -C \"$CUSTOM_PATH\" p4 p4d &&\n+\tsudo xattr -d com.apple.quarantine \"$CUSTOM_PATH/p4\" \"$CUSTOM_PATH/p4d\" 2>/dev/null || true\n \trm helix-core-server.tgz\n \n \tif test -n \"$CC_PACKAGE\"\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex d882250db5..4cce854bad 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -340,10 +340,6 @@ ubuntu-*)\n \t# image.\n \t# Keep that in mind when you encounter a broken OS X build!\n \texport LINUX_GIT_LFS_VERSION=\"1.5.2\"\n-\n-\tP4_PATH=\"$HOME/custom/p4\"\n-\tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n-\texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n macos-*)\n \tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n@@ -351,12 +347,12 @@ macos-*)\n \tthen\n \t\tMAKEFLAGS=\"$MAKEFLAGS APPLE_COMMON_CRYPTO_SHA1=Yes\"\n \tfi\n-\n-\tP4_PATH=\"$HOME/custom/p4\"\n-\texport PATH=\"$P4_PATH:$PATH\"\n \t;;\n esac\n \n+CUSTOM_PATH=\"$HOME/path\"\n+export PATH=\"$CUSTOM_PATH:$PATH\"\n+\n case \"$jobname\" in\n linux32)\n \tCC=gcc\n-- \n2.44.GIT\n\n"},{"id":"492770","messageId":"ef61b578da84dbd8087bf22c43029acf6983e547.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 07/13] ci: fix setup of custom path for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:10:31Z","receivedAt":"2024-04-11T09:10:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Part of \"install-dependencies.sh\" is to install some binaries required\nfor tests into a custom directory that gets added to the PATH. This\ndirectory is located at \"$HOME/path\" and thus depends on the current\nuser that the script executes as.\n\nThis creates problems for GitLab CI, which installs dependencies as the\nroot user, but runs tests as a separate, unprivileged user. As their\nrespective home directories are different, we will end up using two\ndifferent custom path directories. Consequently, the unprivileged user\nwill not be able eto find the binaries that were setu up as root user.\n\nFix this issue by allowing CI to override the custom path, which allows\nGitLab to set up a constant value that isn't derived from \"$HOME\".\n---\n .gitlab-ci.yml | 2 ++\n ci/lib.sh | 2 +-\n 2 files changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex c0fa2fe90b..3a0ef4d4d4 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -9,6 +9,8 @@ workflow:\n \n test:linux:\n image: $image\n+ variables:\n+ CUSTOM_PATH: \"/custom\"\n before_script:\n - ./ci/install-docker-dependencies.sh\n script:\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 4cce854bad..473a2d0348 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -350,7 +350,7 @@ macos-*)\n \t;;\n esac\n \n-CUSTOM_PATH=\"$HOME/path\"\n+CUSTOM_PATH=\"${CUSTOM_PATH:-$HOME/path}\"\n export PATH=\"$CUSTOM_PATH:$PATH\"\n \n case \"$jobname\" in\n-- \n2.44.GIT\n\n"},{"id":"492771","messageId":"7748f87f8cd23826adbbfbaa7616d8b5be346c42.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 08/13] ci: merge scripts which install dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:10:36Z","receivedAt":"2024-04-11T09:10:40Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different scripts which install dependencies, one for\ndockerized jobs and one for non-dockerized ones. Naturally, these\nscripts have quite some duplication. Furthermore, either of these\nscripts is missing some test dependencies that the respective other\nscript has, thus reducing test coverage.\n\nMerge those two scripts such that there is a single source of truth for\ntest dependencies, only.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 2 +-\n .gitlab-ci.yml | 4 +--\n ci/install-dependencies.sh | 32 ++++++++++++++++++---\n ci/install-docker-dependencies.sh | 46 -------------------------------\n 4 files changed, 31 insertions(+), 53 deletions(-)\n delete mode 100755 ci/install-docker-dependencies.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 71cd4e5486..5838986895 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -359,7 +359,7 @@ jobs:\n if: matrix.vector.jobname != 'linux32'\n - uses: actions/checkout@v1 # cannot be upgraded because Node.js Actions aren't supported in this container\n if: matrix.vector.jobname == 'linux32'\n- - run: ci/install-docker-dependencies.sh\n+ - run: ci/install-dependencies.sh\n - run: ci/run-build-and-tests.sh\n - name: print test failures\n if: failure() && env.FAILED_TEST_ARTIFACTS != ''\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 3a0ef4d4d4..ba65f50aac 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -12,7 +12,7 @@ test:linux:\n variables:\n CUSTOM_PATH: \"/custom\"\n before_script:\n- - ./ci/install-docker-dependencies.sh\n+ - ./ci/install-dependencies.sh\n script:\n - useradd builder --create-home\n - chown -R builder \"${CI_PROJECT_DIR}\"\n@@ -100,7 +100,7 @@ static-analysis:\n variables:\n jobname: StaticAnalysis\n before_script:\n- - ./ci/install-docker-dependencies.sh\n+ - ./ci/install-dependencies.sh\n script:\n - ./ci/run-static-analysis.sh\n - ./ci/check-directional-formatting.bash\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex bafe37f2d1..e673797115 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -5,6 +5,8 @@\n \n . ${0%/*}/lib.sh\n \n+begin_group \"Install dependencies\"\n+\n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n \n@@ -20,14 +22,27 @@ then\n fi\n \n case \"$distro\" in\n+alpine-*)\n+\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n+\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n+\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n+\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n+\t;;\n+fedora-*)\n+\tdnf -yq update >/dev/null &&\n+\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\t;;\n ubuntu-*)\n+\t# Required so that apt doesn't wait for user input on certain packages.\n+\texport DEBIAN_FRONTEND=noninteractive\n+\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install \\\n-\t\tlanguage-pack-is libsvn-perl apache2 \\\n-\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev \\\n+\t\tlanguage-pack-is libsvn-perl apache2 cvs cvsps git gnupg subversion \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo \\\n \t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n-\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n-\t\t$CC_PACKAGE $PYTHON_PACKAGE\n+\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \\\n+\t\t${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE\n \n \tmkdir --parents \"$CUSTOM_PATH\"\n \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n@@ -39,6 +54,13 @@ ubuntu-*)\n \t\t-C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n+ubuntu32-*)\n+\tsudo linux32 --32bit i386 sh -c '\n+\t\tapt update >/dev/null &&\n+\t\tapt install -y build-essential libcurl4-openssl-dev \\\n+\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n+\t'\n+\t;;\n macos-*)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n \t# Uncomment this if you want to run perf tests:\n@@ -98,3 +120,5 @@ then\n else\n \techo >&2 \"WARNING: git-lfs wasn't installed, see above for clues why\"\n fi\n+\n+end_group \"Install dependencies\"\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\ndeleted file mode 100755\nindex eb2c9e1eca..0000000000\n--- a/ci/install-docker-dependencies.sh\n+++ /dev/null\n@@ -1,46 +0,0 @@\n-#!/bin/sh\n-#\n-# Install dependencies required to build and test Git inside container\n-#\n-\n-. ${0%/*}/lib.sh\n-\n-begin_group \"Install dependencies\"\n-\n-case \"$jobname\" in\n-linux32)\n-\tlinux32 --32bit i386 sh -c '\n-\t\tapt update >/dev/null &&\n-\t\tapt install -y build-essential libcurl4-openssl-dev \\\n-\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n-\t'\n-\t;;\n-linux-musl)\n-\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n-\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n-\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n-\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n-\t;;\n-linux-*|StaticAnalysis)\n-\t# Required so that apt doesn't wait for user input on certain packages.\n-\texport DEBIAN_FRONTEND=noninteractive\n-\n-\tapt update -q &&\n-\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n-\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n-\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n-\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n-\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n-\n-\tif test \"$jobname\" = StaticAnalysis\n-\tthen\n-\t\tapt install -q -y coccinelle\n-\tfi\n-\t;;\n-pedantic)\n-\tdnf -yq update >/dev/null &&\n-\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n-\t;;\n-esac\n-\n-end_group \"Install dependencies\"\n-- \n2.44.GIT\n\n"},{"id":"492772","messageId":"f7399382f2aa5d17424a065bd72ab2fa915947f0.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 09/13] ci: make Perforce binaries executable for all users","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:10:47Z","receivedAt":"2024-04-11T09:10:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The Perforce binaries are only made executable for the current user. On\nGitLab CI though we execute tests as a different user than \"root\", and\nthus these binaries may not be executable by that test user at all. This\nhas gone unnoticed so far because those binaries are optional -- in case\nthey don't exist we simply skip over tests requiring them.\n\nFix the setup so that we set the executable bits for all users.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex e673797115..0b9bb686d8 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -47,7 +47,7 @@ ubuntu-*)\n \tmkdir --parents \"$CUSTOM_PATH\"\n \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n \t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\tchmod u+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n+\tchmod a+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n \n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n-- \n2.44.GIT\n\n"},{"id":"492773","messageId":"b835ff8b78e74ac7ff5ffd69aa0ba65576f76331.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 10/13] ci: install JGit dependency","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:10:54Z","receivedAt":"2024-04-11T09:10:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have some tests in t5310 that use JGit to verify that bitmaps can be\nread both by Git and by JGit. We do not execute these tests in our CI\njobs though because we don't make JGit available there. Consequently,\nthe tests basically bitrot because almost nobody is ever going to have\nJGit in their path.\n\nInstall JGit to plug this test gap.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 15 ++++++++++++++-\n 1 file changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 0b9bb686d8..c196e56762 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -9,6 +9,7 @@ begin_group \"Install dependencies\"\n \n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n+JGITWHENCE=https://repo.eclipse.org/content/groups/releases//org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh\n \n # Make sudo a no-op and execute the command directly when running as root.\n # While using sudo would be fine on most platforms when we are root already,\n@@ -39,7 +40,7 @@ ubuntu-*)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install \\\n \t\tlanguage-pack-is libsvn-perl apache2 cvs cvsps git gnupg subversion \\\n-\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo default-jre \\\n \t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n \t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \\\n \t\t${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE\n@@ -53,6 +54,9 @@ ubuntu-*)\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n \t\t-C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n+\n+\twget --quiet \"$JGITWHENCE\" --output-document=\"$CUSTOM_PATH/jgit\"\n+\tchmod a+x \"$CUSTOM_PATH/jgit\"\n \t;;\n ubuntu32-*)\n \tsudo linux32 --32bit i386 sh -c '\n@@ -113,6 +117,7 @@ then\n else\n \techo >&2 \"WARNING: perforce wasn't installed, see above for clues why\"\n fi\n+\n if type git-lfs >/dev/null 2>&1\n then\n \techo \"$(tput setaf 6)Git-LFS Version$(tput sgr0)\"\n@@ -121,4 +126,12 @@ else\n \techo >&2 \"WARNING: git-lfs wasn't installed, see above for clues why\"\n fi\n \n+if type jgit >/dev/null 2>&1\n+then\n+\techo \"$(tput setaf 6)JGit Version$(tput sgr0)\"\n+\tjgit version\n+else\n+\techo >&2 \"WARNING: JGit wasn't installed, see above for clues why\"\n+fi\n+\n end_group \"Install dependencies\"\n-- \n2.44.GIT\n\n"},{"id":"492774","messageId":"7136c8b6c21457515118c5084a715fed92b427e8.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 11/13] t06xx: always execute backend-specific tests","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:11:00Z","receivedAt":"2024-04-11T09:11:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The tests in t06xx exercise specific ref formats. Next to probing some\nbasic functionality, these tests also exercise other low-level details\nspecific to the format. Those tests are only executed though in case\n`GIT_TEST_DEFAULT_REF_FORMAT` is set to the ref format of the respective\nbackend-under-test.\n\nIdeally, we would run the full test matrix for ref formats such that our\ncomplete test suite is executed with every supported format on every\nsupported platform. This is quite an expensive undertaking though, and\nthus we only execute e.g. the \"reftable\" tests on macOS and Linux. As a\nresult, we basically have no test coverage for the \"reftable\" format at\nall on other platforms like Windows.\n\nAdapt these tests so that they override `GIT_TEST_DEFAULT_REF_FORMAT`,\nwhich means that they'll always execute. This increases test coverage on\nplatforms that don't run the full test matrix, which at least gives us\nsome basic test coverage on those platforms for the \"reftable\" format.\n\nThis of course comes at the cost of running those tests multiple times\non platforms where we do run the full test matrix. But arguably, this is\na good thing because it will also cause us to e.g. run those tests with\nthe address sanitizer and other non-standard parameters.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0600-reffiles-backend.sh | 8 ++------\n t/t0601-reffiles-pack-refs.sh | 9 +++------\n t/t0610-reftable-basics.sh | 9 +++------\n 3 files changed, 8 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t0600-reffiles-backend.sh b/t/t0600-reffiles-backend.sh\nindex 64214340e7..a390cffc80 100755\n--- a/t/t0600-reffiles-backend.sh\n+++ b/t/t0600-reffiles-backend.sh\n@@ -4,16 +4,12 @@ test_description='Test reffiles backend'\n \n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n-if ! test_have_prereq REFFILES\n-then\n-\tskip_all='skipping reffiles specific tests'\n-\ttest_done\n-fi\n-\n test_expect_success 'setup' '\n \tgit commit --allow-empty -m Initial &&\n \tC=$(git rev-parse HEAD) &&\ndiff --git a/t/t0601-reffiles-pack-refs.sh b/t/t0601-reffiles-pack-refs.sh\nindex 7d4ab0b91a..60a544b8ee 100755\n--- a/t/t0601-reffiles-pack-refs.sh\n+++ b/t/t0601-reffiles-pack-refs.sh\n@@ -9,18 +9,15 @@ test_description='git pack-refs should not change the branch semantic\n This test runs git pack-refs and git show-ref and checks that the branch\n semantic is still the same.\n '\n+\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n-if ! test_have_prereq REFFILES\n-then\n-\tskip_all='skipping reffiles specific tests'\n-\ttest_done\n-fi\n-\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 931d888bbb..fd0ddb96ae 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -4,17 +4,14 @@\n #\n \n test_description='reftable basics'\n+\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n . ./test-lib.sh\n \n-if ! test_have_prereq REFTABLE\n-then\n-\tskip_all='skipping reftable tests; set GIT_TEST_DEFAULT_REF_FORMAT=reftable'\n-\ttest_done\n-fi\n-\n INVALID_OID=$(test_oid 001)\n \n test_expect_success 'init: creates basic reftable structures' '\n-- \n2.44.GIT\n\n"},{"id":"492775","messageId":"cf4ee9c427b45491b191fa1d4bc06cdd7844467b.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 12/13] t0610: fix non-portable variable assignment","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:11:05Z","receivedAt":"2024-04-11T09:11:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Older versions of the Dash shell fail to parse `local var=val`\nassignments in some cases when `val` is unquoted. Such failures can be\nobserved e.g. with Ubuntu 20.04 and older, which has a Dash version that\nstill has this bug.\n\nSuch an assignment has been introduced in t0610. The issue wasn't\ndetected for a while because this test used to only run when the\nGIT_TEST_DEFAULT_REF_FORMAT environment variable was set to \"reftable\".\nWe have dropped that requirement now though, meaning that it runs\nunconditionally, including on jobs which use such older versions of\nUbuntu.\n\nWe have worked around such issues in the past, e.g. in ebee5580ca\n(parallel-checkout: avoid dash local bug in tests, 2021-06-06), by\nquoting the `val` side. Apply the same fix to t0610.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0610-reftable-basics.sh | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex fd0ddb96ae..b6e67724ce 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -78,9 +78,9 @@ test_expect_success 'init: reinitializing reftable with files backend fails' '\n '\n \n test_expect_perms () {\n-\tlocal perms=\"$1\"\n-\tlocal file=\"$2\"\n-\tlocal actual=$(ls -l \"$file\") &&\n+\tlocal perms=\"$1\" &&\n+\tlocal file=\"$2\" &&\n+\tlocal actual=\"$(ls -l \"$file\")\" &&\n \n \tcase \"$actual\" in\n \t$perms*)\n-- \n2.44.GIT\n\n"},{"id":"492776","messageId":"a9cd20eebcd2407604b9d534088197696f74bba1.1712825204.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712825204.git.ps@pks.im","subject":"[PATCH v3 13/13] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T09:11:09Z","receivedAt":"2024-04-11T09:11:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While the reftable format is a recent introduction in Git, JGit already\nknows to read and write reftables since 2017. Given the complexity of\nthe format there is a very real risk of incompatibilities between those\ntwo implementations, which is something that we really want to avoid.\n\nAdd some basic tests that verify that reftables written by Git and JGit\ncan be read by the respective other implementation. For now this test\nsuite is rather small, only covering basic functionality. But it serves\nas a good starting point and can be extended over time.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0612-reftable-jgit-compatibility.sh | 132 +++++++++++++++++++++++++\n 1 file changed, 132 insertions(+)\n create mode 100755 t/t0612-reftable-jgit-compatibility.sh\n\ndiff --git a/t/t0612-reftable-jgit-compatibility.sh b/t/t0612-reftable-jgit-compatibility.sh\nnew file mode 100755\nindex 0000000000..d0d7e80b49\n--- /dev/null\n+++ b/t/t0612-reftable-jgit-compatibility.sh\n@@ -0,0 +1,132 @@\n+#!/bin/sh\n+\n+test_description='reftables are compatible with JGit'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+# JGit does not support the 'link' DIRC extension.\n+GIT_TEST_SPLIT_INDEX=0\n+export GIT_TEST_SPLIT_INDEX\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq JGIT\n+then\n+\tskip_all='skipping reftable JGit tests; JGit is not present in PATH'\n+\ttest_done\n+fi\n+\n+if ! test_have_prereq SHA1\n+then\n+\tskip_all='skipping reftable JGit tests; JGit does not support SHA256 reftables'\n+\ttest_done\n+fi\n+\n+test_commit_jgit () {\n+\ttouch \"$1\" &&\n+\tjgit add \"$1\" &&\n+\tjgit commit -m \"$1\"\n+}\n+\n+test_same_refs () {\n+\tgit show-ref --head >cgit.actual &&\n+\tjgit show-ref >jgit-tabs.actual &&\n+\ttr \"\\t\" \" \" jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_same_ref () {\n+\tgit rev-parse \"$1\" >cgit.actual &&\n+\tjgit rev-parse \"$1\" >jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_same_reflog () {\n+\tgit reflog \"$*\" >cgit.actual &&\n+\tjgit reflog \"$*\" >jgit-newline.actual &&\n+\tsed '/^$/d' jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_expect_success 'CGit repository can be read by JGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\ttest_same_refs &&\n+\t\ttest_same_ref HEAD &&\n+\t\ttest_same_reflog HEAD\n+\t)\n+'\n+\n+test_expect_success 'JGit repository can be read by CGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tjgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttouch file &&\n+\t\tjgit add file &&\n+\t\tjgit commit -m \"initial commit\" &&\n+\n+\t\t# Note that we must convert the ref storage after we have\n+\t\t# written the default branch. Otherwise JGit will end up with\n+\t\t# no HEAD at all.\n+\t\tjgit convert-ref-storage --format=reftable &&\n+\n+\t\ttest_same_refs &&\n+\t\ttest_same_ref HEAD &&\n+\t\t# Interestingly, JGit cannot read its own reflog here. CGit can\n+\t\t# though.\n+\t\tprintf \"%s HEAD@{0}: commit (initial): initial commit\" \"$(git rev-parse --short HEAD)\" >expect &&\n+\t\tgit reflog HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'mixed writes from JGit and CGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit_jgit B &&\n+\t\ttest_commit C &&\n+\t\ttest_commit_jgit D &&\n+\n+\t\ttest_same_refs &&\n+\t\ttest_same_ref HEAD &&\n+\t\ttest_same_reflog HEAD\n+\t)\n+'\n+\n+test_expect_success 'JGit can read multi-level index' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\tawk \"\n+\t\t BEGIN {\n+\t\t\tprint \\\"start\\\";\n+\t\t\tfor (i = 0; i < 10000; i++)\n+\t\t\t printf \\\"create refs/heads/branch-%d HEAD\\n\\\", i;\n+\t\t\tprint \\\"commit\\\";\n+\t\t }\n+\t\t\" >input &&\n+\t\tgit update-ref --stdin wrote:\n> Part of \"install-dependencies.sh\" is to install some binaries required\n> for tests into a custom directory that gets added to the PATH. This\n> directory is located at \"$HOME/path\" and thus depends on the current\n> user that the script executes as.\n>\n> This creates problems for GitLab CI, which installs dependencies as the\n> root user, but runs tests as a separate, unprivileged user. As their\n> respective home directories are different, we will end up using two\n> different custom path directories. Consequently, the unprivileged user\n> will not be able eto find the binaries that were setu up as root user.\n\ns/eto/to/\ns/setu/setup/\n\n> Fix this issue by allowing CI to override the custom path, which allows\n> GitLab to set up a constant value that isn't derived from \"$HOME\".\n> ---\n\nMissing sign-off.\n"},{"id":"492780","messageId":"Zhe642d8hMr63H76@tanuki","threadId":"61266","inReplyTo":"CAPig+cR19_xdKx84xQ+7c4Cp6GzeOeHSit7-ujJJQ6gE=q+f_w@mail.gmail.com","subject":"Re: [PATCH v3 07/13] ci: fix setup of custom path for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-11T10:26:43Z","receivedAt":"2024-04-11T10:26:51Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Apr 11, 2024 at 06:06:05AM -0400, Eric Sunshine wrote:\n> On Thu, Apr 11, 2024 at 5:10 AM Patrick Steinhardt wrote:\n> > Part of \"install-dependencies.sh\" is to install some binaries required\n> > for tests into a custom directory that gets added to the PATH. This\n> > directory is located at \"$HOME/path\" and thus depends on the current\n> > user that the script executes as.\n> >\n> > This creates problems for GitLab CI, which installs dependencies as the\n> > root user, but runs tests as a separate, unprivileged user. As their\n> > respective home directories are different, we will end up using two\n> > different custom path directories. Consequently, the unprivileged user\n> > will not be able eto find the binaries that were setu up as root user.\n> \n> s/eto/to/\n> s/setu/setup/\n\nUgh. Well, to my excuse, my keyboard is a bit wonky recently.\n\n> > Fix this issue by allowing CI to override the custom path, which allows\n> > GitLab to set up a constant value that isn't derived from \"$HOME\".\n> > ---\n> \n> Missing sign-off.\n\nIndeed. Sorry, I'm somewhat distracted today.\n\nPatrick\n"},{"id":"492829","messageId":"cover.1712896868.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712235356.git.ps@pks.im","subject":"[PATCH v4 00/13] t: exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:43:52Z","receivedAt":"2024-04-12T04:44:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nhere's the (hopefully last) version of my patch series that introduces\nGit/JGit compatibility tests for the reftable format. Only a single\ncommit changed, where I fixed two typos and added a missing signoff.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (13):\n ci: rename \"runs_on_pool\" to \"distro\"\n ci: expose distro name in dockerized GitHub jobs\n ci: skip sudo when we are already root\n ci: drop duplicate package installation for \"linux-gcc-default\"\n ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"\n ci: merge custom PATH directories\n ci: fix setup of custom path for GitLab CI\n ci: merge scripts which install dependencies\n ci: make Perforce binaries executable for all users\n ci: install JGit dependency\n t06xx: always execute backend-specific tests\n t0610: fix non-portable variable assignment\n t0612: add tests to exercise Git/JGit reftable compatibility\n\n .github/workflows/main.yml | 8 +-\n .gitlab-ci.yml | 6 +-\n ci/install-dependencies.sh | 101 +++++++++++++------\n ci/install-docker-dependencies.sh | 46 ---------\n ci/lib.sh | 14 +--\n t/t0600-reffiles-backend.sh | 8 +-\n t/t0601-reffiles-pack-refs.sh | 9 +-\n t/t0610-reftable-basics.sh | 15 ++-\n t/t0612-reftable-jgit-compatibility.sh | 132 +++++++++++++++++++++++++\n 9 files changed, 229 insertions(+), 110 deletions(-)\n delete mode 100755 ci/install-docker-dependencies.sh\n create mode 100755 t/t0612-reftable-jgit-compatibility.sh\n\nRange-diff against v3:\n 1: 46502bbe22 = 1: 99b0b60359 ci: rename \"runs_on_pool\" to \"distro\"\n 2: d076ed9857 = 2: e1d4e1320d ci: expose distro name in dockerized GitHub jobs\n 3: cc0c29052f = 3: 44a0e07600 ci: skip sudo when we are already root\n 4: 803f5020e0 = 4: 8e58ce38d9 ci: drop duplicate package installation for \"linux-gcc-default\"\n 5: d2745e9438 = 5: b8f56a5e67 ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"\n 6: 61f108d954 = 6: ce4f0c766c ci: merge custom PATH directories\n 7: ef61b578da ! 7: 9fc462b578 ci: fix setup of custom path for GitLab CI\n @@ Commit message\n root user, but runs tests as a separate, unprivileged user. As their\n respective home directories are different, we will end up using two\n different custom path directories. Consequently, the unprivileged user\n - will not be able eto find the binaries that were setu up as root user.\n + will not be able to find the binaries that were set up as root user.\n \n Fix this issue by allowing CI to override the custom path, which allows\n GitLab to set up a constant value that isn't derived from \"$HOME\".\n \n + Signed-off-by: Patrick Steinhardt \n +\n ## .gitlab-ci.yml ##\n @@ .gitlab-ci.yml: workflow:\n \n 8: 7748f87f8c = 8: e7a17d57e7 ci: merge scripts which install dependencies\n 9: f7399382f2 = 9: 720d5a4682 ci: make Perforce binaries executable for all users\n10: b835ff8b78 = 10: 77f6d6ecaa ci: install JGit dependency\n11: 7136c8b6c2 = 11: acf0c28550 t06xx: always execute backend-specific tests\n12: cf4ee9c427 = 12: a9b71e8eea t0610: fix non-portable variable assignment\n13: a9cd20eebc = 13: 218c694d2e t0612: add tests to exercise Git/JGit reftable compatibility\n\nbase-commit: 436d4e5b14df49870a897f64fe92c0ddc7017e4c\n-- \n2.44.GIT\n\n"},{"id":"492830","messageId":"99b0b60359c1466fd291e890b8403d9cbfd706c8.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 01/13] ci: rename \"runs_on_pool\" to \"distro\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:43:57Z","receivedAt":"2024-04-12T04:44:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"runs_on_pool\" environment variable is used by our CI scripts to\ndistinguish the different kinds of operating systems. It is quite\nspecific to GitHub Actions though and not really a descriptive name.\n\nRename the variable to \"distro\" to clarify its intent.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 2 +-\n ci/install-dependencies.sh | 2 +-\n ci/lib.sh | 4 ++--\n 3 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 3428773b09..684ef5c00d 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -303,7 +303,7 @@ jobs:\n CC: ${{matrix.vector.cc}}\n CC_PACKAGE: ${{matrix.vector.cc_package}}\n jobname: ${{matrix.vector.jobname}}\n- runs_on_pool: ${{matrix.vector.pool}}\n+ distro: ${{matrix.vector.pool}}\n runs-on: ${{matrix.vector.pool}}\n steps:\n - uses: actions/checkout@v4\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex b4e22de3cb..7d247b5ef4 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -11,7 +11,7 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n-case \"$runs_on_pool\" in\n+case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 0a73fc7bd1..d882250db5 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -279,7 +279,7 @@ then\n \n \tcache_dir=\"$HOME/none\"\n \n-\truns_on_pool=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n+\tdistro=$(echo \"$CI_JOB_IMAGE\" | tr : -)\n \tJOBS=$(nproc)\n else\n \techo \"Could not identify CI type\" >&2\n@@ -318,7 +318,7 @@ export DEFAULT_TEST_TARGET=prove\n export GIT_TEST_CLONE_2GB=true\n export SKIP_DASHED_BUILT_INS=YesPlease\n \n-case \"$runs_on_pool\" in\n+case \"$distro\" in\n ubuntu-*)\n \tif test \"$jobname\" = \"linux-gcc-default\"\n \tthen\n-- \n2.44.GIT\n\n"},{"id":"492831","messageId":"e1d4e1320d8834de3c37e3c0e9a6e4682aa79a0b.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 02/13] ci: expose distro name in dockerized GitHub jobs","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:02Z","receivedAt":"2024-04-12T04:44:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Expose a distro name in dockerized jobs. This will be used in a\nsubsequent commit where we merge the installation scripts for dockerized\nand non-dockerized jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 684ef5c00d..71cd4e5486 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -342,12 +342,16 @@ jobs:\n vector:\n - jobname: linux-musl\n image: alpine\n+ distro: alpine-latest\n - jobname: linux32\n image: daald/ubuntu32:xenial\n+ distro: ubuntu32-16.04\n - jobname: pedantic\n image: fedora\n+ distro: fedora-latest\n env:\n jobname: ${{matrix.vector.jobname}}\n+ distro: ${{matrix.vector.distro}}\n runs-on: ubuntu-latest\n container: ${{matrix.vector.image}}\n steps:\n-- \n2.44.GIT\n\n"},{"id":"492832","messageId":"44a0e0760010a585a63e61b922783157227d3fb8.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 03/13] ci: skip sudo when we are already root","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:07Z","receivedAt":"2024-04-12T04:44:11Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Our \"install-dependencies.sh\" script is executed by non-dockerized jobs\nto install dependencies. These jobs don't run with \"root\" permissions,\nbut with a separate user. Consequently, we need to use sudo(8) there to\nelevate permissions when installing packages.\n\nWe're about to merge \"install-docker-dependencies.sh\" into that script\nthough, and our Docker containers do run as \"root\". Using sudo(8) is\nthus unnecessary there, even though it would be harmless. On some images\nlike Alpine Linux though there is no sudo(8) available by default, which\nwould consequently break the build.\n\nAdapt the script to make \"sudo\" a no-op when running as \"root\" user.\nThis allows us to easily reuse the script for our dockerized jobs.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 11 +++++++++++\n 1 file changed, 11 insertions(+)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7d247b5ef4..7dfd3e50ed 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -11,6 +11,17 @@ UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n+# Make sudo a no-op and execute the command directly when running as root.\n+# While using sudo would be fine on most platforms when we are root already,\n+# some platforms like e.g. Alpine Linux do not have sudo available by default\n+# and would thus break.\n+if test \"$(id -u)\" -eq 0\n+then\n+\tsudo () {\n+\t\t\"$@\"\n+\t}\n+fi\n+\n case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n-- \n2.44.GIT\n\n"},{"id":"492833","messageId":"8e58ce38d9a8a9465d172d3ef091a7a7cdde3393.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 04/13] ci: drop duplicate package installation for \"linux-gcc-default\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:12Z","receivedAt":"2024-04-12T04:44:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"linux-gcc-default\" job installs common Ubuntu packages. This is\nalready done in the distro-specific switch, so we basically duplicate\nthe effort here.\n\nDrop the duplicate package installations and inline the variable that\ncontains those common packages.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 17 ++++++++---------\n 1 file changed, 8 insertions(+), 9 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7dfd3e50ed..fad53aac96 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -7,9 +7,6 @@\n \n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n-UBUNTU_COMMON_PKGS=\"make libssl-dev libcurl4-openssl-dev libexpat-dev\n- tcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl\n- libemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl\"\n \n # Make sudo a no-op and execute the command directly when running as root.\n # While using sudo would be fine on most platforms when we are root already,\n@@ -25,8 +22,13 @@ fi\n case \"$distro\" in\n ubuntu-*)\n \tsudo apt-get -q update\n-\tsudo apt-get -q -y install language-pack-is libsvn-perl apache2 \\\n-\t\t$UBUNTU_COMMON_PKGS $CC_PACKAGE $PYTHON_PACKAGE\n+\tsudo apt-get -q -y install \\\n+\t\tlanguage-pack-is libsvn-perl apache2 \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev \\\n+\t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n+\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n+\t\t$CC_PACKAGE $PYTHON_PACKAGE\n+\n \tmkdir --parents \"$P4_PATH\"\n \tpushd \"$P4_PATH\"\n \t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n@@ -34,6 +36,7 @@ ubuntu-*)\n \t\tchmod u+x p4d\n \t\tchmod u+x p4\n \tpopd\n+\n \tmkdir --parents \"$GIT_LFS_PATH\"\n \tpushd \"$GIT_LFS_PATH\"\n \t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n@@ -83,10 +86,6 @@ Documentation)\n \ttest -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n \tsudo gem install --version 1.5.8 asciidoctor\n \t;;\n-linux-gcc-default)\n-\tsudo apt-get -q update\n-\tsudo apt-get -q -y install $UBUNTU_COMMON_PKGS\n-\t;;\n esac\n \n if type p4d >/dev/null 2>&1 && type p4 >/dev/null 2>&1\n-- \n2.44.GIT\n\n"},{"id":"492834","messageId":"b8f56a5e67495f4f2efe6ee991eeac386f600dc2.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 05/13] ci: convert \"install-dependencies.sh\" to use \"/bin/sh\"","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:17Z","receivedAt":"2024-04-12T04:44:22Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're about to merge the \"install-docker-dependencies.sh\" script into\n\"install-dependencies.sh\". This will also move our Alpine-based jobs\nover to use the latter script. This script uses the Bash shell though,\nwhich is not available by default on Alpine Linux.\n\nRefactor \"install-dependencies.sh\" to use \"/bin/sh\" instead of Bash.\nThis requires us to get rid of the pushd/popd invocations, which are\nreplaced by some more elaborate commands that download or extract\nexecutables right to where they are needed.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 29 ++++++++++++-----------------\n 1 file changed, 12 insertions(+), 17 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex fad53aac96..2d6af876d6 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -1,4 +1,4 @@\n-#!/usr/bin/env bash\n+#!/bin/sh\n #\n # Install dependencies required to build and test Git on Linux and macOS\n #\n@@ -30,19 +30,15 @@ ubuntu-*)\n \t\t$CC_PACKAGE $PYTHON_PACKAGE\n \n \tmkdir --parents \"$P4_PATH\"\n-\tpushd \"$P4_PATH\"\n-\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4d\"\n-\t\twget --quiet \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\t\tchmod u+x p4d\n-\t\tchmod u+x p4\n-\tpopd\n+\twget --quiet --directory-prefix=\"$P4_PATH\" \\\n+\t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n+\tchmod u+x \"$P4_PATH/p4d\" \"$P4_PATH/p4\"\n \n \tmkdir --parents \"$GIT_LFS_PATH\"\n-\tpushd \"$GIT_LFS_PATH\"\n-\t\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\t\ttar --extract --gunzip --file \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n-\t\tcp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .\n-\tpopd\n+\twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n+\ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n+\t\t-C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n+\trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n macos-*)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n@@ -53,11 +49,10 @@ macos-*)\n \tbrew link --force gettext\n \n \tmkdir -p \"$P4_PATH\"\n-\tpushd \"$P4_PATH\"\n-\t\twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n-\t\ttar -xf helix-core-server.tgz &&\n-\t\tsudo xattr -d com.apple.quarantine p4 p4d 2>/dev/null || true\n-\tpopd\n+\twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n+\ttar -xf helix-core-server.tgz -C \"$P4_PATH\" p4 p4d &&\n+\tsudo xattr -d com.apple.quarantine \"$P4_PATH/p4\" \"$P4_PATH/p4d\" 2>/dev/null || true\n+\trm helix-core-server.tgz\n \n \tif test -n \"$CC_PACKAGE\"\n \tthen\n-- \n2.44.GIT\n\n"},{"id":"492835","messageId":"ce4f0c766c46d1d8f8ac3920e5efe832c5dd71fb.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 06/13] ci: merge custom PATH directories","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:22Z","receivedAt":"2024-04-12T04:44:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're downloading various executables required by our tests. Each of\nthese executables goes into its own directory, which is then appended to\nthe PATH variable. Consequently, whenever we add a new dependency and\nthus a new directory, we would have to adapt to this change in several\nplaces.\n\nRefactor this to instead put all binaries into a single directory.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 15 +++++++--------\n ci/lib.sh | 10 +++-------\n 2 files changed, 10 insertions(+), 15 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 2d6af876d6..bafe37f2d1 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -29,15 +29,14 @@ ubuntu-*)\n \t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n \t\t$CC_PACKAGE $PYTHON_PACKAGE\n \n-\tmkdir --parents \"$P4_PATH\"\n-\twget --quiet --directory-prefix=\"$P4_PATH\" \\\n+\tmkdir --parents \"$CUSTOM_PATH\"\n+\twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n \t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\tchmod u+x \"$P4_PATH/p4d\" \"$P4_PATH/p4\"\n+\tchmod u+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n \n-\tmkdir --parents \"$GIT_LFS_PATH\"\n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n-\t\t-C \"$GIT_LFS_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n+\t\t-C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n macos-*)\n@@ -48,10 +47,10 @@ macos-*)\n \tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \n-\tmkdir -p \"$P4_PATH\"\n+\tmkdir -p \"$CUSTOM_PATH\"\n \twget -q \"$P4WHENCE/bin.macosx1015x86_64/helix-core-server.tgz\" &&\n-\ttar -xf helix-core-server.tgz -C \"$P4_PATH\" p4 p4d &&\n-\tsudo xattr -d com.apple.quarantine \"$P4_PATH/p4\" \"$P4_PATH/p4d\" 2>/dev/null || true\n+\ttar -xf helix-core-server.tgz -C \"$CUSTOM_PATH\" p4 p4d &&\n+\tsudo xattr -d com.apple.quarantine \"$CUSTOM_PATH/p4\" \"$CUSTOM_PATH/p4d\" 2>/dev/null || true\n \trm helix-core-server.tgz\n \n \tif test -n \"$CC_PACKAGE\"\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex d882250db5..4cce854bad 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -340,10 +340,6 @@ ubuntu-*)\n \t# image.\n \t# Keep that in mind when you encounter a broken OS X build!\n \texport LINUX_GIT_LFS_VERSION=\"1.5.2\"\n-\n-\tP4_PATH=\"$HOME/custom/p4\"\n-\tGIT_LFS_PATH=\"$HOME/custom/git-lfs\"\n-\texport PATH=\"$GIT_LFS_PATH:$P4_PATH:$PATH\"\n \t;;\n macos-*)\n \tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=$(which python3)\"\n@@ -351,12 +347,12 @@ macos-*)\n \tthen\n \t\tMAKEFLAGS=\"$MAKEFLAGS APPLE_COMMON_CRYPTO_SHA1=Yes\"\n \tfi\n-\n-\tP4_PATH=\"$HOME/custom/p4\"\n-\texport PATH=\"$P4_PATH:$PATH\"\n \t;;\n esac\n \n+CUSTOM_PATH=\"$HOME/path\"\n+export PATH=\"$CUSTOM_PATH:$PATH\"\n+\n case \"$jobname\" in\n linux32)\n \tCC=gcc\n-- \n2.44.GIT\n\n"},{"id":"492836","messageId":"9fc462b578b74ecc34aaeda5a1c8bcf3e0ba66ff.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 07/13] ci: fix setup of custom path for GitLab CI","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:27Z","receivedAt":"2024-04-12T04:44:31Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Part of \"install-dependencies.sh\" is to install some binaries required\nfor tests into a custom directory that gets added to the PATH. This\ndirectory is located at \"$HOME/path\" and thus depends on the current\nuser that the script executes as.\n\nThis creates problems for GitLab CI, which installs dependencies as the\nroot user, but runs tests as a separate, unprivileged user. As their\nrespective home directories are different, we will end up using two\ndifferent custom path directories. Consequently, the unprivileged user\nwill not be able to find the binaries that were set up as root user.\n\nFix this issue by allowing CI to override the custom path, which allows\nGitLab to set up a constant value that isn't derived from \"$HOME\".\n\nSigned-off-by: Patrick Steinhardt \n---\n .gitlab-ci.yml | 2 ++\n ci/lib.sh | 2 +-\n 2 files changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex c0fa2fe90b..3a0ef4d4d4 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -9,6 +9,8 @@ workflow:\n \n test:linux:\n image: $image\n+ variables:\n+ CUSTOM_PATH: \"/custom\"\n before_script:\n - ./ci/install-docker-dependencies.sh\n script:\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 4cce854bad..473a2d0348 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -350,7 +350,7 @@ macos-*)\n \t;;\n esac\n \n-CUSTOM_PATH=\"$HOME/path\"\n+CUSTOM_PATH=\"${CUSTOM_PATH:-$HOME/path}\"\n export PATH=\"$CUSTOM_PATH:$PATH\"\n \n case \"$jobname\" in\n-- \n2.44.GIT\n\n"},{"id":"492837","messageId":"e7a17d57e7013a5aab27045c26d48260256ba9a0.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 08/13] ci: merge scripts which install dependencies","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:32Z","receivedAt":"2024-04-12T04:44:37Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have two different scripts which install dependencies, one for\ndockerized jobs and one for non-dockerized ones. Naturally, these\nscripts have quite some duplication. Furthermore, either of these\nscripts is missing some test dependencies that the respective other\nscript has, thus reducing test coverage.\n\nMerge those two scripts such that there is a single source of truth for\ntest dependencies, only.\n\nSigned-off-by: Patrick Steinhardt \n---\n .github/workflows/main.yml | 2 +-\n .gitlab-ci.yml | 4 +--\n ci/install-dependencies.sh | 32 ++++++++++++++++++---\n ci/install-docker-dependencies.sh | 46 -------------------------------\n 4 files changed, 31 insertions(+), 53 deletions(-)\n delete mode 100755 ci/install-docker-dependencies.sh\n\ndiff --git a/.github/workflows/main.yml b/.github/workflows/main.yml\nindex 71cd4e5486..5838986895 100644\n--- a/.github/workflows/main.yml\n+++ b/.github/workflows/main.yml\n@@ -359,7 +359,7 @@ jobs:\n if: matrix.vector.jobname != 'linux32'\n - uses: actions/checkout@v1 # cannot be upgraded because Node.js Actions aren't supported in this container\n if: matrix.vector.jobname == 'linux32'\n- - run: ci/install-docker-dependencies.sh\n+ - run: ci/install-dependencies.sh\n - run: ci/run-build-and-tests.sh\n - name: print test failures\n if: failure() && env.FAILED_TEST_ARTIFACTS != ''\ndiff --git a/.gitlab-ci.yml b/.gitlab-ci.yml\nindex 3a0ef4d4d4..ba65f50aac 100644\n--- a/.gitlab-ci.yml\n+++ b/.gitlab-ci.yml\n@@ -12,7 +12,7 @@ test:linux:\n variables:\n CUSTOM_PATH: \"/custom\"\n before_script:\n- - ./ci/install-docker-dependencies.sh\n+ - ./ci/install-dependencies.sh\n script:\n - useradd builder --create-home\n - chown -R builder \"${CI_PROJECT_DIR}\"\n@@ -100,7 +100,7 @@ static-analysis:\n variables:\n jobname: StaticAnalysis\n before_script:\n- - ./ci/install-docker-dependencies.sh\n+ - ./ci/install-dependencies.sh\n script:\n - ./ci/run-static-analysis.sh\n - ./ci/check-directional-formatting.bash\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex bafe37f2d1..e673797115 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -5,6 +5,8 @@\n \n . ${0%/*}/lib.sh\n \n+begin_group \"Install dependencies\"\n+\n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n \n@@ -20,14 +22,27 @@ then\n fi\n \n case \"$distro\" in\n+alpine-*)\n+\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n+\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n+\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n+\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n+\t;;\n+fedora-*)\n+\tdnf -yq update >/dev/null &&\n+\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n+\t;;\n ubuntu-*)\n+\t# Required so that apt doesn't wait for user input on certain packages.\n+\texport DEBIAN_FRONTEND=noninteractive\n+\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install \\\n-\t\tlanguage-pack-is libsvn-perl apache2 \\\n-\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev \\\n+\t\tlanguage-pack-is libsvn-perl apache2 cvs cvsps git gnupg subversion \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo \\\n \t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n-\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl \\\n-\t\t$CC_PACKAGE $PYTHON_PACKAGE\n+\t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \\\n+\t\t${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE\n \n \tmkdir --parents \"$CUSTOM_PATH\"\n \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n@@ -39,6 +54,13 @@ ubuntu-*)\n \t\t-C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \t;;\n+ubuntu32-*)\n+\tsudo linux32 --32bit i386 sh -c '\n+\t\tapt update >/dev/null &&\n+\t\tapt install -y build-essential libcurl4-openssl-dev \\\n+\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n+\t'\n+\t;;\n macos-*)\n \texport HOMEBREW_NO_AUTO_UPDATE=1 HOMEBREW_NO_INSTALL_CLEANUP=1\n \t# Uncomment this if you want to run perf tests:\n@@ -98,3 +120,5 @@ then\n else\n \techo >&2 \"WARNING: git-lfs wasn't installed, see above for clues why\"\n fi\n+\n+end_group \"Install dependencies\"\ndiff --git a/ci/install-docker-dependencies.sh b/ci/install-docker-dependencies.sh\ndeleted file mode 100755\nindex eb2c9e1eca..0000000000\n--- a/ci/install-docker-dependencies.sh\n+++ /dev/null\n@@ -1,46 +0,0 @@\n-#!/bin/sh\n-#\n-# Install dependencies required to build and test Git inside container\n-#\n-\n-. ${0%/*}/lib.sh\n-\n-begin_group \"Install dependencies\"\n-\n-case \"$jobname\" in\n-linux32)\n-\tlinux32 --32bit i386 sh -c '\n-\t\tapt update >/dev/null &&\n-\t\tapt install -y build-essential libcurl4-openssl-dev \\\n-\t\t\tlibssl-dev libexpat-dev gettext python >/dev/null\n-\t'\n-\t;;\n-linux-musl)\n-\tapk add --update shadow sudo build-base curl-dev openssl-dev expat-dev gettext \\\n-\t\tpcre2-dev python3 musl-libintl perl-utils ncurses \\\n-\t\tapache2 apache2-http2 apache2-proxy apache2-ssl apache2-webdav apr-util-dbd_sqlite3 \\\n-\t\tbash cvs gnupg perl-cgi perl-dbd-sqlite >/dev/null\n-\t;;\n-linux-*|StaticAnalysis)\n-\t# Required so that apt doesn't wait for user input on certain packages.\n-\texport DEBIAN_FRONTEND=noninteractive\n-\n-\tapt update -q &&\n-\tapt install -q -y sudo git make language-pack-is libsvn-perl apache2 libssl-dev \\\n-\t\tlibcurl4-openssl-dev libexpat-dev tcl tk gettext zlib1g-dev \\\n-\t\tperl-modules liberror-perl libauthen-sasl-perl libemail-valid-perl \\\n-\t\tlibdbd-sqlite3-perl libio-socket-ssl-perl libnet-smtp-ssl-perl ${CC_PACKAGE:-${CC:-gcc}} \\\n-\t\tapache2 cvs cvsps gnupg libcgi-pm-perl subversion\n-\n-\tif test \"$jobname\" = StaticAnalysis\n-\tthen\n-\t\tapt install -q -y coccinelle\n-\tfi\n-\t;;\n-pedantic)\n-\tdnf -yq update >/dev/null &&\n-\tdnf -yq install make gcc findutils diffutils perl python3 gettext zlib-devel expat-devel openssl-devel curl-devel pcre2-devel >/dev/null\n-\t;;\n-esac\n-\n-end_group \"Install dependencies\"\n-- \n2.44.GIT\n\n"},{"id":"492838","messageId":"720d5a46822231ce8cce26a0704d5a3e6ec95b58.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 09/13] ci: make Perforce binaries executable for all users","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:37Z","receivedAt":"2024-04-12T04:44:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The Perforce binaries are only made executable for the current user. On\nGitLab CI though we execute tests as a different user than \"root\", and\nthus these binaries may not be executable by that test user at all. This\nhas gone unnoticed so far because those binaries are optional -- in case\nthey don't exist we simply skip over tests requiring them.\n\nFix the setup so that we set the executable bits for all users.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex e673797115..0b9bb686d8 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -47,7 +47,7 @@ ubuntu-*)\n \tmkdir --parents \"$CUSTOM_PATH\"\n \twget --quiet --directory-prefix=\"$CUSTOM_PATH\" \\\n \t\t\"$P4WHENCE/bin.linux26x86_64/p4d\" \"$P4WHENCE/bin.linux26x86_64/p4\"\n-\tchmod u+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n+\tchmod a+x \"$CUSTOM_PATH/p4d\" \"$CUSTOM_PATH/p4\"\n \n \twget --quiet \"$LFSWHENCE/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n-- \n2.44.GIT\n\n"},{"id":"492839","messageId":"77f6d6ecaac037ad03b8e93ea392e0270bad9e45.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 10/13] ci: install JGit dependency","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:43Z","receivedAt":"2024-04-12T04:44:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We have some tests in t5310 that use JGit to verify that bitmaps can be\nread both by Git and by JGit. We do not execute these tests in our CI\njobs though because we don't make JGit available there. Consequently,\nthe tests basically bitrot because almost nobody is ever going to have\nJGit in their path.\n\nInstall JGit to plug this test gap.\n\nSigned-off-by: Patrick Steinhardt \n---\n ci/install-dependencies.sh | 15 ++++++++++++++-\n 1 file changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 0b9bb686d8..c196e56762 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -9,6 +9,7 @@ begin_group \"Install dependencies\"\n \n P4WHENCE=https://cdist2.perforce.com/perforce/r21.2\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n+JGITWHENCE=https://repo.eclipse.org/content/groups/releases//org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh\n \n # Make sudo a no-op and execute the command directly when running as root.\n # While using sudo would be fine on most platforms when we are root already,\n@@ -39,7 +40,7 @@ ubuntu-*)\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install \\\n \t\tlanguage-pack-is libsvn-perl apache2 cvs cvsps git gnupg subversion \\\n-\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo \\\n+\t\tmake libssl-dev libcurl4-openssl-dev libexpat-dev wget sudo default-jre \\\n \t\ttcl tk gettext zlib1g-dev perl-modules liberror-perl libauthen-sasl-perl \\\n \t\tlibemail-valid-perl libio-socket-ssl-perl libnet-smtp-ssl-perl libdbd-sqlite3-perl libcgi-pm-perl \\\n \t\t${CC_PACKAGE:-${CC:-gcc}} $PYTHON_PACKAGE\n@@ -53,6 +54,9 @@ ubuntu-*)\n \ttar -xzf \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\" \\\n \t\t-C \"$CUSTOM_PATH\" --strip-components=1 \"git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs\"\n \trm \"git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz\"\n+\n+\twget --quiet \"$JGITWHENCE\" --output-document=\"$CUSTOM_PATH/jgit\"\n+\tchmod a+x \"$CUSTOM_PATH/jgit\"\n \t;;\n ubuntu32-*)\n \tsudo linux32 --32bit i386 sh -c '\n@@ -113,6 +117,7 @@ then\n else\n \techo >&2 \"WARNING: perforce wasn't installed, see above for clues why\"\n fi\n+\n if type git-lfs >/dev/null 2>&1\n then\n \techo \"$(tput setaf 6)Git-LFS Version$(tput sgr0)\"\n@@ -121,4 +126,12 @@ else\n \techo >&2 \"WARNING: git-lfs wasn't installed, see above for clues why\"\n fi\n \n+if type jgit >/dev/null 2>&1\n+then\n+\techo \"$(tput setaf 6)JGit Version$(tput sgr0)\"\n+\tjgit version\n+else\n+\techo >&2 \"WARNING: JGit wasn't installed, see above for clues why\"\n+fi\n+\n end_group \"Install dependencies\"\n-- \n2.44.GIT\n\n"},{"id":"492840","messageId":"acf0c285506fe7ba275b08cdaf6b2245ec66b565.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 11/13] t06xx: always execute backend-specific tests","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:48Z","receivedAt":"2024-04-12T04:44:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The tests in t06xx exercise specific ref formats. Next to probing some\nbasic functionality, these tests also exercise other low-level details\nspecific to the format. Those tests are only executed though in case\n`GIT_TEST_DEFAULT_REF_FORMAT` is set to the ref format of the respective\nbackend-under-test.\n\nIdeally, we would run the full test matrix for ref formats such that our\ncomplete test suite is executed with every supported format on every\nsupported platform. This is quite an expensive undertaking though, and\nthus we only execute e.g. the \"reftable\" tests on macOS and Linux. As a\nresult, we basically have no test coverage for the \"reftable\" format at\nall on other platforms like Windows.\n\nAdapt these tests so that they override `GIT_TEST_DEFAULT_REF_FORMAT`,\nwhich means that they'll always execute. This increases test coverage on\nplatforms that don't run the full test matrix, which at least gives us\nsome basic test coverage on those platforms for the \"reftable\" format.\n\nThis of course comes at the cost of running those tests multiple times\non platforms where we do run the full test matrix. But arguably, this is\na good thing because it will also cause us to e.g. run those tests with\nthe address sanitizer and other non-standard parameters.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0600-reffiles-backend.sh | 8 ++------\n t/t0601-reffiles-pack-refs.sh | 9 +++------\n t/t0610-reftable-basics.sh | 9 +++------\n 3 files changed, 8 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t0600-reffiles-backend.sh b/t/t0600-reffiles-backend.sh\nindex 64214340e7..a390cffc80 100755\n--- a/t/t0600-reffiles-backend.sh\n+++ b/t/t0600-reffiles-backend.sh\n@@ -4,16 +4,12 @@ test_description='Test reffiles backend'\n \n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n-if ! test_have_prereq REFFILES\n-then\n-\tskip_all='skipping reffiles specific tests'\n-\ttest_done\n-fi\n-\n test_expect_success 'setup' '\n \tgit commit --allow-empty -m Initial &&\n \tC=$(git rev-parse HEAD) &&\ndiff --git a/t/t0601-reffiles-pack-refs.sh b/t/t0601-reffiles-pack-refs.sh\nindex 7d4ab0b91a..60a544b8ee 100755\n--- a/t/t0601-reffiles-pack-refs.sh\n+++ b/t/t0601-reffiles-pack-refs.sh\n@@ -9,18 +9,15 @@ test_description='git pack-refs should not change the branch semantic\n This test runs git pack-refs and git show-ref and checks that the branch\n semantic is still the same.\n '\n+\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=files\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n-if ! test_have_prereq REFFILES\n-then\n-\tskip_all='skipping reffiles specific tests'\n-\ttest_done\n-fi\n-\n test_expect_success 'enable reflogs' '\n \tgit config core.logallrefupdates true\n '\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex 931d888bbb..fd0ddb96ae 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -4,17 +4,14 @@\n #\n \n test_description='reftable basics'\n+\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n \n . ./test-lib.sh\n \n-if ! test_have_prereq REFTABLE\n-then\n-\tskip_all='skipping reftable tests; set GIT_TEST_DEFAULT_REF_FORMAT=reftable'\n-\ttest_done\n-fi\n-\n INVALID_OID=$(test_oid 001)\n \n test_expect_success 'init: creates basic reftable structures' '\n-- \n2.44.GIT\n\n"},{"id":"492841","messageId":"a9b71e8eeabee42378312c860c7af49aee641491.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 12/13] t0610: fix non-portable variable assignment","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:53Z","receivedAt":"2024-04-12T04:44:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Older versions of the Dash shell fail to parse `local var=val`\nassignments in some cases when `val` is unquoted. Such failures can be\nobserved e.g. with Ubuntu 20.04 and older, which has a Dash version that\nstill has this bug.\n\nSuch an assignment has been introduced in t0610. The issue wasn't\ndetected for a while because this test used to only run when the\nGIT_TEST_DEFAULT_REF_FORMAT environment variable was set to \"reftable\".\nWe have dropped that requirement now though, meaning that it runs\nunconditionally, including on jobs which use such older versions of\nUbuntu.\n\nWe have worked around such issues in the past, e.g. in ebee5580ca\n(parallel-checkout: avoid dash local bug in tests, 2021-06-06), by\nquoting the `val` side. Apply the same fix to t0610.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0610-reftable-basics.sh | 6 +++---\n 1 file changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh\nindex fd0ddb96ae..b6e67724ce 100755\n--- a/t/t0610-reftable-basics.sh\n+++ b/t/t0610-reftable-basics.sh\n@@ -78,9 +78,9 @@ test_expect_success 'init: reinitializing reftable with files backend fails' '\n '\n \n test_expect_perms () {\n-\tlocal perms=\"$1\"\n-\tlocal file=\"$2\"\n-\tlocal actual=$(ls -l \"$file\") &&\n+\tlocal perms=\"$1\" &&\n+\tlocal file=\"$2\" &&\n+\tlocal actual=\"$(ls -l \"$file\")\" &&\n \n \tcase \"$actual\" in\n \t$perms*)\n-- \n2.44.GIT\n\n"},{"id":"492842","messageId":"218c694d2e1230b7b4f231e83feafc945820a26c.1712896869.git.ps@pks.im","threadId":"61266","inReplyTo":"cover.1712896868.git.ps@pks.im","subject":"[PATCH v4 13/13] t0612: add tests to exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-04-12T04:44:58Z","receivedAt":"2024-04-12T04:45:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"While the reftable format is a recent introduction in Git, JGit already\nknows to read and write reftables since 2017. Given the complexity of\nthe format there is a very real risk of incompatibilities between those\ntwo implementations, which is something that we really want to avoid.\n\nAdd some basic tests that verify that reftables written by Git and JGit\ncan be read by the respective other implementation. For now this test\nsuite is rather small, only covering basic functionality. But it serves\nas a good starting point and can be extended over time.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t0612-reftable-jgit-compatibility.sh | 132 +++++++++++++++++++++++++\n 1 file changed, 132 insertions(+)\n create mode 100755 t/t0612-reftable-jgit-compatibility.sh\n\ndiff --git a/t/t0612-reftable-jgit-compatibility.sh b/t/t0612-reftable-jgit-compatibility.sh\nnew file mode 100755\nindex 0000000000..d0d7e80b49\n--- /dev/null\n+++ b/t/t0612-reftable-jgit-compatibility.sh\n@@ -0,0 +1,132 @@\n+#!/bin/sh\n+\n+test_description='reftables are compatible with JGit'\n+\n+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n+GIT_TEST_DEFAULT_REF_FORMAT=reftable\n+export GIT_TEST_DEFAULT_REF_FORMAT\n+\n+# JGit does not support the 'link' DIRC extension.\n+GIT_TEST_SPLIT_INDEX=0\n+export GIT_TEST_SPLIT_INDEX\n+\n+. ./test-lib.sh\n+\n+if ! test_have_prereq JGIT\n+then\n+\tskip_all='skipping reftable JGit tests; JGit is not present in PATH'\n+\ttest_done\n+fi\n+\n+if ! test_have_prereq SHA1\n+then\n+\tskip_all='skipping reftable JGit tests; JGit does not support SHA256 reftables'\n+\ttest_done\n+fi\n+\n+test_commit_jgit () {\n+\ttouch \"$1\" &&\n+\tjgit add \"$1\" &&\n+\tjgit commit -m \"$1\"\n+}\n+\n+test_same_refs () {\n+\tgit show-ref --head >cgit.actual &&\n+\tjgit show-ref >jgit-tabs.actual &&\n+\ttr \"\\t\" \" \" jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_same_ref () {\n+\tgit rev-parse \"$1\" >cgit.actual &&\n+\tjgit rev-parse \"$1\" >jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_same_reflog () {\n+\tgit reflog \"$*\" >cgit.actual &&\n+\tjgit reflog \"$*\" >jgit-newline.actual &&\n+\tsed '/^$/d' jgit.actual &&\n+\ttest_cmp cgit.actual jgit.actual\n+}\n+\n+test_expect_success 'CGit repository can be read by JGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\t\ttest_commit A &&\n+\t\ttest_same_refs &&\n+\t\ttest_same_ref HEAD &&\n+\t\ttest_same_reflog HEAD\n+\t)\n+'\n+\n+test_expect_success 'JGit repository can be read by CGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tjgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttouch file &&\n+\t\tjgit add file &&\n+\t\tjgit commit -m \"initial commit\" &&\n+\n+\t\t# Note that we must convert the ref storage after we have\n+\t\t# written the default branch. Otherwise JGit will end up with\n+\t\t# no HEAD at all.\n+\t\tjgit convert-ref-storage --format=reftable &&\n+\n+\t\ttest_same_refs &&\n+\t\ttest_same_ref HEAD &&\n+\t\t# Interestingly, JGit cannot read its own reflog here. CGit can\n+\t\t# though.\n+\t\tprintf \"%s HEAD@{0}: commit (initial): initial commit\" \"$(git rev-parse --short HEAD)\" >expect &&\n+\t\tgit reflog HEAD >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'mixed writes from JGit and CGit' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\ttest_commit_jgit B &&\n+\t\ttest_commit C &&\n+\t\ttest_commit_jgit D &&\n+\n+\t\ttest_same_refs &&\n+\t\ttest_same_ref HEAD &&\n+\t\ttest_same_reflog HEAD\n+\t)\n+'\n+\n+test_expect_success 'JGit can read multi-level index' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit A &&\n+\t\tawk \"\n+\t\t BEGIN {\n+\t\t\tprint \\\"start\\\";\n+\t\t\tfor (i = 0; i < 10000; i++)\n+\t\t\t printf \\\"create refs/heads/branch-%d HEAD\\n\\\", i;\n+\t\t\tprint \\\"commit\\\";\n+\t\t }\n+\t\t\" >input &&\n+\t\tgit update-ref --stdin Hi,\n> \n> here's the (hopefully last) version of my patch series that introduces\n> Git/JGit compatibility tests for the reftable format. Only a single\n> commit changed, where I fixed two typos and added a missing signoff.\n\nWith 35e293e6 (Merge branch 'ps/ci-test-with-jgit' into next,\n2024-05-01), I've noticed that a subset of the `test:linux` GitLab CI\njobs have started failing on next.\n\nhttps://gitlab.com/gitlab-org/git/-/pipelines/1277877090\n\nI'll take a look.\n\n-Justin\n"},{"id":"494042","messageId":"ZjUxY7kqgy-iLIzG@ncase","threadId":"61266","inReplyTo":"x7v4d4siixs3wllzycibxtgsqipnwohhyxg72zvcxboucshd2t@won3yobvixbu","subject":"Re: [PATCH v4 00/13] t: exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-03T18:48:03Z","receivedAt":"2024-05-03T18:48:10Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 03, 2024 at 01:42:32PM -0500, Justin Tobler wrote:\n> On 24/04/12 06:43AM, Patrick Steinhardt wrote:\n> > Hi,\n> > \n> > here's the (hopefully last) version of my patch series that introduces\n> > Git/JGit compatibility tests for the reftable format. Only a single\n> > commit changed, where I fixed two typos and added a missing signoff.\n> \n> With 35e293e6 (Merge branch 'ps/ci-test-with-jgit' into next,\n> 2024-05-01), I've noticed that a subset of the `test:linux` GitLab CI\n> jobs have started failing on next.\n> \n> https://gitlab.com/gitlab-org/git/-/pipelines/1277877090\n> \n> I'll take a look.\n\nAre you sure it's related to this merge? All failures are on\n\"ubuntu:latest\", and the complaints are about the \"python2\" package\nbeing missing. Given the recent release of Ubuntu 24.04, maybe the root\ncause is that \"python2\" does not exist there anymore?\n\nPatrick\n"},{"id":"494044","messageId":"ZjUzjhN78lqYtXqT@ncase","threadId":"61266","inReplyTo":"ZjUxY7kqgy-iLIzG@ncase","subject":"Re: [PATCH v4 00/13] t: exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-03T18:57:18Z","receivedAt":"2024-05-03T18:57:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 03, 2024 at 08:48:03PM +0200, Patrick Steinhardt wrote:\n> On Fri, May 03, 2024 at 01:42:32PM -0500, Justin Tobler wrote:\n> > On 24/04/12 06:43AM, Patrick Steinhardt wrote:\n> > > Hi,\n> > > \n> > > here's the (hopefully last) version of my patch series that introduces\n> > > Git/JGit compatibility tests for the reftable format. Only a single\n> > > commit changed, where I fixed two typos and added a missing signoff.\n> > \n> > With 35e293e6 (Merge branch 'ps/ci-test-with-jgit' into next,\n> > 2024-05-01), I've noticed that a subset of the `test:linux` GitLab CI\n> > jobs have started failing on next.\n> > \n> > https://gitlab.com/gitlab-org/git/-/pipelines/1277877090\n> > \n> > I'll take a look.\n> \n> Are you sure it's related to this merge? All failures are on\n> \"ubuntu:latest\", and the complaints are about the \"python2\" package\n> being missing. Given the recent release of Ubuntu 24.04, maybe the root\n> cause is that \"python2\" does not exist there anymore?\n> \n> Patrick\n\nMaybe we should do something like below patch. Basically, we start to\nacknowledge the fact that Python 2 is end of life and always use Python\n3 on ubuntu:latest. We might go even further than that and only use\nPython 2 on ubuntu:20.04 and slowly phase out support for it.\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 473a2d0348..3967a5af85 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -325,11 +325,18 @@ ubuntu-*)\n \t\tbreak\n \tfi\n \n-\tPYTHON_PACKAGE=python2\n-\tif test \"$jobname\" = linux-gcc\n-\tthen\n+\tcase \"$distro\" in\n+\tubuntu-latest)\n \t\tPYTHON_PACKAGE=python3\n-\tfi\n+\t\t;;\n+\t*)\n+\t\tPYTHON_PACKAGE=python2\n+\t\tif test \"$jobname\" = linux-gcc\n+\t\tthen\n+\t\t\tPYTHON_PACKAGE=python3\n+\t\tfi\n+\t\t;;\n+\tesac\n \tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/$PYTHON_PACKAGE\"\n \n \texport GIT_TEST_HTTPD=true\n\nPatrick\n"},{"id":"494054","messageId":"2hhwttin6gvs2munwpl2zhnm2573bm5vklbk6mcwxbgc7z5sog@3dygnrnnnpna","threadId":"61266","inReplyTo":"ZjUzjhN78lqYtXqT@ncase","subject":"Re: [PATCH v4 00/13] t: exercise Git/JGit reftable compatibility","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-05-03T19:35:46Z","receivedAt":"2024-05-03T19:37:05Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/05/03 08:57PM, Patrick Steinhardt wrote:\n> On Fri, May 03, 2024 at 08:48:03PM +0200, Patrick Steinhardt wrote:\n> > On Fri, May 03, 2024 at 01:42:32PM -0500, Justin Tobler wrote:\n> > > On 24/04/12 06:43AM, Patrick Steinhardt wrote:\n> > > > Hi,\n> > > > \n> > > > here's the (hopefully last) version of my patch series that introduces\n> > > > Git/JGit compatibility tests for the reftable format. Only a single\n> > > > commit changed, where I fixed two typos and added a missing signoff.\n> > > \n> > > With 35e293e6 (Merge branch 'ps/ci-test-with-jgit' into next,\n> > > 2024-05-01), I've noticed that a subset of the `test:linux` GitLab CI\n> > > jobs have started failing on next.\n> > > \n> > > https://gitlab.com/gitlab-org/git/-/pipelines/1277877090\n> > > \n> > > I'll take a look.\n> > \n> > Are you sure it's related to this merge? All failures are on\n> > \"ubuntu:latest\", and the complaints are about the \"python2\" package\n> > being missing. Given the recent release of Ubuntu 24.04, maybe the root\n> > cause is that \"python2\" does not exist there anymore?\n\nYa, you are right. That appears to be the problem here.\n\n> Maybe we should do something like below patch. Basically, we start to\n> acknowledge the fact that Python 2 is end of life and always use Python\n> 3 on ubuntu:latest. We might go even further than that and only use\n> Python 2 on ubuntu:20.04 and slowly phase out support for it.\n> \n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 473a2d0348..3967a5af85 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -325,11 +325,18 @@ ubuntu-*)\n> \t\tbreak\n> \tfi\n> \n> -\tPYTHON_PACKAGE=python2\n> -\tif test \"$jobname\" = linux-gcc\n> -\tthen\n> +\tcase \"$distro\" in\n> +\tubuntu-latest)\n> \t\tPYTHON_PACKAGE=python3\n> -\tfi\n> +\t\t;;\n> +\t*)\n> +\t\tPYTHON_PACKAGE=python2\n> +\t\tif test \"$jobname\" = linux-gcc\n> +\t\tthen\n> +\t\t\tPYTHON_PACKAGE=python3\n> +\t\tfi\n> +\t\t;;\n> +\tesac\n> \tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/$PYTHON_PACKAGE\"\n> \n> \texport GIT_TEST_HTTPD=true\n\nThis seems reasonable to me :)\n\n-Justin\n\n"},{"id":"494058","messageId":"ZjU_1PJnvmGhtckO@ncase","threadId":"61266","inReplyTo":"2hhwttin6gvs2munwpl2zhnm2573bm5vklbk6mcwxbgc7z5sog@3dygnrnnnpna","subject":"Re: [PATCH v4 00/13] t: exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-03T19:49:40Z","receivedAt":"2024-05-03T19:49:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, May 03, 2024 at 02:35:46PM -0500, Justin Tobler wrote:\n> On 24/05/03 08:57PM, Patrick Steinhardt wrote:\n> > On Fri, May 03, 2024 at 08:48:03PM +0200, Patrick Steinhardt wrote:\n> > > On Fri, May 03, 2024 at 01:42:32PM -0500, Justin Tobler wrote:\n> > > > On 24/04/12 06:43AM, Patrick Steinhardt wrote:\n[snip]\n> > Maybe we should do something like below patch. Basically, we start to\n> > acknowledge the fact that Python 2 is end of life and always use Python\n> > 3 on ubuntu:latest. We might go even further than that and only use\n> > Python 2 on ubuntu:20.04 and slowly phase out support for it.\n> > \n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index 473a2d0348..3967a5af85 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -325,11 +325,18 @@ ubuntu-*)\n> > \t\tbreak\n> > \tfi\n> > \n> > -\tPYTHON_PACKAGE=python2\n> > -\tif test \"$jobname\" = linux-gcc\n> > -\tthen\n> > +\tcase \"$distro\" in\n> > +\tubuntu-latest)\n> > \t\tPYTHON_PACKAGE=python3\n> > -\tfi\n> > +\t\t;;\n> > +\t*)\n> > +\t\tPYTHON_PACKAGE=python2\n> > +\t\tif test \"$jobname\" = linux-gcc\n> > +\t\tthen\n> > +\t\t\tPYTHON_PACKAGE=python3\n> > +\t\tfi\n> > +\t\t;;\n> > +\tesac\n> > \tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/$PYTHON_PACKAGE\"\n> > \n> > \texport GIT_TEST_HTTPD=true\n> \n> This seems reasonable to me :)\n\nPlease feel free to adopt and adapt this fix. I probably shouldn't be\nreading mails at this time of the day in the first place :)\n\nPatrick\n"},{"id":"494082","messageId":"enfz4bzeg5m3odkbhrzwjfgah3cd5fl333rgos3sz3aju33y4y@do3yczu7a6hc","threadId":"61266","inReplyTo":"ZjU_1PJnvmGhtckO@ncase","subject":"Re: [PATCH v4 00/13] t: exercise Git/JGit reftable compatibility","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2024-05-04T17:32:31Z","receivedAt":"2024-05-04T17:33:51Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 24/05/03 09:49PM, Patrick Steinhardt wrote:\n> On Fri, May 03, 2024 at 02:35:46PM -0500, Justin Tobler wrote:\n> > On 24/05/03 08:57PM, Patrick Steinhardt wrote:\n> > > On Fri, May 03, 2024 at 08:48:03PM +0200, Patrick Steinhardt wrote:\n> > > > On Fri, May 03, 2024 at 01:42:32PM -0500, Justin Tobler wrote:\n> > > > > On 24/04/12 06:43AM, Patrick Steinhardt wrote:\n> [snip]\n> > > Maybe we should do something like below patch. Basically, we start to\n> > > acknowledge the fact that Python 2 is end of life and always use Python\n> > > 3 on ubuntu:latest. We might go even further than that and only use\n> > > Python 2 on ubuntu:20.04 and slowly phase out support for it.\n> > > \n> > > diff --git a/ci/lib.sh b/ci/lib.sh\n> > > index 473a2d0348..3967a5af85 100755\n> > > --- a/ci/lib.sh\n> > > +++ b/ci/lib.sh\n> > > @@ -325,11 +325,18 @@ ubuntu-*)\n> > > \t\tbreak\n> > > \tfi\n> > > \n> > > -\tPYTHON_PACKAGE=python2\n> > > -\tif test \"$jobname\" = linux-gcc\n> > > -\tthen\n> > > +\tcase \"$distro\" in\n> > > +\tubuntu-latest)\n> > > \t\tPYTHON_PACKAGE=python3\n> > > -\tfi\n> > > +\t\t;;\n> > > +\t*)\n> > > +\t\tPYTHON_PACKAGE=python2\n> > > +\t\tif test \"$jobname\" = linux-gcc\n> > > +\t\tthen\n> > > +\t\t\tPYTHON_PACKAGE=python3\n> > > +\t\tfi\n> > > +\t\t;;\n> > > +\tesac\n> > > \tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/$PYTHON_PACKAGE\"\n> > > \n> > > \texport GIT_TEST_HTTPD=true\n> > \n> > This seems reasonable to me :)\n> \n> Please feel free to adopt and adapt this fix. I probably shouldn't be\n> reading mails at this time of the day in the first place :)\n\nThanks, I'll give it a go!\n\n-Justin\n\n"},{"id":"494111","messageId":"ZjhwYNWO-tdqcUyt@tanuki","threadId":"61266","inReplyTo":"enfz4bzeg5m3odkbhrzwjfgah3cd5fl333rgos3sz3aju33y4y@do3yczu7a6hc","subject":"Re: [PATCH v4 00/13] t: exercise Git/JGit reftable compatibility","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-05-06T05:53:36Z","receivedAt":"2024-05-06T05:53:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, May 04, 2024 at 12:32:31PM -0500, Justin Tobler wrote:\n> On 24/05/03 09:49PM, Patrick Steinhardt wrote:\n> > On Fri, May 03, 2024 at 02:35:46PM -0500, Justin Tobler wrote:\n> > > On 24/05/03 08:57PM, Patrick Steinhardt wrote:\n> > > > On Fri, May 03, 2024 at 08:48:03PM +0200, Patrick Steinhardt wrote:\n> > > > > On Fri, May 03, 2024 at 01:42:32PM -0500, Justin Tobler wrote:\n> > > > > > On 24/04/12 06:43AM, Patrick Steinhardt wrote:\n> > [snip]\n> > > > Maybe we should do something like below patch. Basically, we start to\n> > > > acknowledge the fact that Python 2 is end of life and always use Python\n> > > > 3 on ubuntu:latest. We might go even further than that and only use\n> > > > Python 2 on ubuntu:20.04 and slowly phase out support for it.\n> > > > \n> > > > diff --git a/ci/lib.sh b/ci/lib.sh\n> > > > index 473a2d0348..3967a5af85 100755\n> > > > --- a/ci/lib.sh\n> > > > +++ b/ci/lib.sh\n> > > > @@ -325,11 +325,18 @@ ubuntu-*)\n> > > > \t\tbreak\n> > > > \tfi\n> > > > \n> > > > -\tPYTHON_PACKAGE=python2\n> > > > -\tif test \"$jobname\" = linux-gcc\n> > > > -\tthen\n> > > > +\tcase \"$distro\" in\n> > > > +\tubuntu-latest)\n> > > > \t\tPYTHON_PACKAGE=python3\n> > > > -\tfi\n> > > > +\t\t;;\n> > > > +\t*)\n> > > > +\t\tPYTHON_PACKAGE=python2\n> > > > +\t\tif test \"$jobname\" = linux-gcc\n> > > > +\t\tthen\n> > > > +\t\t\tPYTHON_PACKAGE=python3\n> > > > +\t\tfi\n> > > > +\t\t;;\n> > > > +\tesac\n> > > > \tMAKEFLAGS=\"$MAKEFLAGS PYTHON_PATH=/usr/bin/$PYTHON_PACKAGE\"\n> > > > \n> > > > \texport GIT_TEST_HTTPD=true\n> > > \n> > > This seems reasonable to me :)\n> > \n> > Please feel free to adopt and adapt this fix. I probably shouldn't be\n> > reading mails at this time of the day in the first place :)\n> \n> Thanks, I'll give it a go!\n> \n> -Justin\n\nI've sent out a fix for this via [1] now so that we can hopefully\nfast-track this.\n\nPatrick\n\n[1]: https://lore.kernel.org/git/cb8cefc20f373a3516695e7cbee975132553ea95.1714973381.git.ps@pks.im/T/#u\n"}]}