{"thread":{"id":"49270","subject":"[PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","startedAt":"2018-09-03T21:10:32Z","lastAt":"2021-03-26T19:55:26Z","messageCount":225,"participants":["Johannes Schindelin via GitGitGadget","Eric Sunshine","Johannes Schindelin","Luke Diamand","Sebastian Schuberth","Stefan Beller","Jeff King","Taylor Blau","Ævar Arnfjörð Bjarmason","Duy Nguyen","Christian Couder","Junio C Hamano","SZEDER Gábor"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"357257","messageId":"pull.31.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":null,"subject":"[PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-03T21:10:28Z","receivedAt":"2018-09-03T21:10:32Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"For a long time already, we have Git's source code continuously tested via\nTravis CI, see e.g. https://travis-ci.org/git/git/builds/421738884. It has\nserved us well, and more and more developers actually pay attention and\nbenefit from the testing this gives us.\n\nIt is also an invaluable tool for contributors who can validate their code\ncontributions via PRs on GitHub, e.g. to verify that their tests do actually\nrun on macOS (i.e. with the BSD family of Unix tools instead of the GNU\none).\n\nThe one sad part about this is the Windows support. Travis lacks it, and we\nwork around that by using Visual Studio Team Services (VSTS) indirectly: one\nphase in Travis would trigger a build, wait for its log, and then paste that\nlog.\n\nAs Git's Windows builds (and tests!) take quite a bit of time, Travis often\ntimed out, or somehow the trigger did not work, and for security reasons\n(the Windows builds are performed in a private pool of containers), the\nWindows builds are completely disabled for Pull Requests on GitHub.\n\nOne might ask why we did not use Visual Studio Team Services directly. There\nwere a couple of reasons for that:\n\n * most notably, VSTS's build logs could not be viewed anonymously,\n * while VSTS had Linux and Windows agents, it lacked macOS agents,\n * etc\n\nThe main two reasons no longer apply: macOS agents are available now\n[https://docs.microsoft.com/en-us/vsts/release-notes/2018/jul-10-vsts], and\nthere is a limited preview of \"public projects\"\n[https://blogs.msdn.microsoft.com/devops/2018/04/27/vsts-public-projects-limited-preview/]\n, i.e. it is possible to configure a VSTS project so that anybody can view\nthe logs.\n\nI had secured such a public project for Git for Windows already, and I\nrecently also got one for Git. For now, the latter is hooked up with my\npersonal git.git fork on GitHub, but it is my hope that I convince y'all\nthat these VSTS builds are a good idea, and then hook it up with \nhttps://github.com/git/git.\n\nAs a special treat, this patch series adds the ability to present the\noutcome of Git's test suite as JUnit-style .xml files. This allows the VSTS\nbuild to present fun diagrams, trends, and makes it a lot easier to drill\ndown to test failures than before. See for example \nhttps://git.visualstudio.com/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details\n[https://git.visualstudio.com/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details] \n(you can click on the label of the failed test, and then see the detailed\noutput in the right pane).\n\nThis patch series took way more time than I had originally planned, but I\nthink that in particular the advanced display of the test results was worth\nit. Please let me know what you think about this.\n\nJohannes Schindelin (9):\n ci: rename the library of common functions\n ci/lib.sh: encapsulate Travis-specific things\n test-date: add a subcommand to measure times in shell scripts\n tests: optionally write results as JUnit-style .xml\n ci/lib.sh: add support for VSTS CI\n Add a build definition for VSTS\n tests: include detailed trace logs with --write-junit-xml upon failure\n tests: record more stderr with --write-junit-xml in case of failure\n README: add a build badge (status of the VSTS build)\n\n .vsts-ci.yml | 296 +++++++++++++++++++++++++++++++++\n README.md | 2 +\n ci/install-dependencies.sh | 5 +-\n ci/{lib-travisci.sh => lib.sh} | 67 ++++++--\n ci/mount-fileshare.sh | 26 +++\n ci/print-test-failures.sh | 4 +-\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 3 +-\n t/.gitignore | 1 +\n t/helper/test-date.c | 12 ++\n t/test-lib.sh | 142 ++++++++++++++++\n 14 files changed, 544 insertions(+), 22 deletions(-)\n create mode 100644 .vsts-ci.yml\n rename ci/{lib-travisci.sh => lib.sh} (61%)\n create mode 100755 ci/mount-fileshare.sh\n\n\nbase-commit: 2f743933341f276111103550fbf383a34dfcfd38\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-31%2Fdscho%2Fvsts-ci-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-31/dscho/vsts-ci-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/31\n-- \ngitgitgadget\n"},{"id":"357258","messageId":"858b80bfcfada8f8306181c6ca7c131217494c78.1536009027.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"[PATCH 1/9] ci: rename the library of common functions","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-03T21:10:30Z","receivedAt":"2018-09-03T21:10:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe name is hard-coded to reflect that we use Travis CI for continuous\ntesting.\n\nIn the next commits, we will extend this to be able use Visual Studio\nTeam Services, too.\n\nSo let's adjust the name to make it more generic.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 2 +-\n ci/{lib-travisci.sh => lib.sh} | 0\n ci/print-test-failures.sh | 2 +-\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 2 +-\n 8 files changed, 7 insertions(+), 7 deletions(-)\n rename ci/{lib-travisci.sh => lib.sh} (100%)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 75a9fd2475..961064658e 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -3,7 +3,7 @@\n # Install dependencies required to build and test Git on Linux and macOS\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\ndiff --git a/ci/lib-travisci.sh b/ci/lib.sh\nsimilarity index 100%\nrename from ci/lib-travisci.sh\nrename to ci/lib.sh\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex d55460a212..7aef39a2fd 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -3,7 +3,7 @@\n # Print output of failing tests\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n # Tracing executed commands would produce too much noise in the loop below.\n set +x\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 2a5bff4a1c..e28ac2fb9a 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -3,7 +3,7 @@\n # Build and test Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n ln -s \"$cache_dir/.prove\" t/.prove\n \ndiff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh\nindex 21637903ce..751acfcf8a 100755\n--- a/ci/run-linux32-docker.sh\n+++ b/ci/run-linux32-docker.sh\n@@ -3,7 +3,7 @@\n # Download and run Docker image to build and test 32-bit Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n docker pull daald/ubuntu32:xenial\n \ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\nindex 5688f261d0..dc189c7456 100755\n--- a/ci/run-static-analysis.sh\n+++ b/ci/run-static-analysis.sh\n@@ -3,7 +3,7 @@\n # Perform various static code analysis checks\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n make --jobs=2 coccicheck\n \ndiff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh\nindex d99a180e52..a73a4eca0a 100755\n--- a/ci/run-windows-build.sh\n+++ b/ci/run-windows-build.sh\n@@ -6,7 +6,7 @@\n # supported) and a commit hash.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n test $# -ne 2 && echo \"Unexpected number of parameters\" && exit 1\n test -z \"$GFW_CI_TOKEN\" && echo \"GFW_CI_TOKEN not defined\" && exit\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex a20de9ca12..d3cdbac73f 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -3,7 +3,7 @@\n # Perform sanity checks on documentation and build it.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n gem install asciidoctor\n \n-- \ngitgitgadget\n\n"},{"id":"357259","messageId":"064db77669dab6c63a3fca559c0edbecd1f4ef80.1536009027.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"[PATCH 3/9] test-date: add a subcommand to measure times in shell scripts","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-03T21:10:33Z","receivedAt":"2018-09-03T21:10:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIn the next commit, we want to teach Git's test suite to optionally\noutput test results in JUnit-style .xml files. These files contain\ninformation about the time spent. So we need a way to measure time.\n\nWhile we could use `date +%s` for that, this will give us only seconds,\ni.e. very coarse-grained timings.\n\nGNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output),\nbut there is no equivalent in BSD `date` (read: on macOS, we would not\nbe able to obtain precise timings).\n\nSo let's introduce `test-tool date getnanos`, with an optional start\ntime, that outputs preciser values.\n\nGranted, it is a bit pointless to try measuring times accurately in\nshell scripts, certainly to nanosecond precision. But it is better than\nsecond-granularity.\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-date.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex a0837371ab..792a805374 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -7,6 +7,7 @@ static const char *usage_msg = \"\\n\"\n \" test-tool date parse [date]...\\n\"\n \" test-tool date approxidate [date]...\\n\"\n \" test-tool date timestamp [date]...\\n\"\n+\" test-tool date getnanos [start-nanos]\\n\"\n \" test-tool date is64bit\\n\"\n \" test-tool date time_t-is64bit\\n\";\n \n@@ -82,6 +83,15 @@ static void parse_approx_timestamp(const char **argv, struct timeval *now)\n \t}\n }\n \n+static void getnanos(const char **argv, struct timeval *now)\n+{\n+\tdouble seconds = getnanotime() / 1.0e9;\n+\n+\tif (*argv)\n+\t\tseconds -= strtod(*argv, NULL);\n+\tprintf(\"%lf\\n\", seconds);\n+}\n+\n int cmd__date(int argc, const char **argv)\n {\n \tstruct timeval now;\n@@ -108,6 +118,8 @@ int cmd__date(int argc, const char **argv)\n \t\tparse_approxidate(argv+1, &now);\n \telse if (!strcmp(*argv, \"timestamp\"))\n \t\tparse_approx_timestamp(argv+1, &now);\n+\telse if (!strcmp(*argv, \"getnanos\"))\n+\t\tgetnanos(argv+1, &now);\n \telse if (!strcmp(*argv, \"is64bit\"))\n \t\treturn sizeof(timestamp_t) == 8 ? 0 : 1;\n \telse if (!strcmp(*argv, \"time_t-is64bit\"))\n-- \ngitgitgadget\n\n"},{"id":"357260","messageId":"18e6beec5f48ad5a17c7c7309e0507b01a53c2ec.1536009027.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"[PATCH 2/9] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-03T21:10:31Z","receivedAt":"2018-09-03T21:10:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe upcoming patches will allow building git.git via VSTS CI, where\nvariable names and URLs look a bit different than in Travis CI.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 3 ++-\n ci/lib.sh | 44 +++++++++++++++++++++++++++-----------\n ci/print-test-failures.sh | 2 +-\n ci/test-documentation.sh | 1 +\n 4 files changed, 35 insertions(+), 15 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 961064658e..6d92cc1cba 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -28,7 +28,8 @@ osx-clang|osx-gcc)\n \tbrew update --quiet\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\n-\tbrew install git-lfs gettext\n+\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n+\teval brew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \tbrew install caskroom/cask/perforce\n \t;;\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 06970f7213..657ff88672 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,5 +1,26 @@\n # Library of functions shared by all CI scripts\n \n+if test -n \"$TRAVIS_COMMIT\"\n+then\n+\t# We are running within Travis CI\n+\tCI_BRANCH=\"$TRAVIS_BRANCH\"\n+\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n+\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n+\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n+\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n+\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n+\n+\tcache_dir=\"$HOME/travis-cache\"\n+\n+\turl_for_job_id () {\n+\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+fi\n+\n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n \t# at the same commit as the tip of the branch is pushed, and building\n@@ -13,10 +34,10 @@ skip_branch_tip_with_tag () {\n \t# we can skip the build because we won't be skipping a build\n \t# of a tag.\n \n-\tif TAG=$(git describe --exact-match \"$TRAVIS_BRANCH\" 2>/dev/null) &&\n-\t\ttest \"$TAG\" != \"$TRAVIS_BRANCH\"\n+\tif TAG=$(git describe --exact-match \"$CI_BRANCH\" 2>/dev/null) &&\n+\t\ttest \"$TAG\" != \"$CI_BRANCH\"\n \tthen\n-\t\techo \"$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)\"\n+\t\techo \"$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)\"\n \t\texit 0\n \tfi\n }\n@@ -25,7 +46,7 @@ skip_branch_tip_with_tag () {\n # job if we encounter the same tree again and can provide a useful info\n # message.\n save_good_tree () {\n-\techo \"$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID\" >>\"$good_trees_file\"\n+\techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n \t# limit the file size\n \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n \tmv \"$good_trees_file\".tmp \"$good_trees_file\"\n@@ -35,7 +56,7 @@ save_good_tree () {\n # successfully before (e.g. because the branch got rebased, changing only\n # the commit messages).\n skip_good_tree () {\n-\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $TRAVIS_COMMIT^{tree}) \" \"$good_trees_file\")\"\n+\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n \tthen\n \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n \t\t# Continue the build job.\n@@ -45,18 +66,18 @@ skip_good_tree () {\n \techo \"$good_tree_info\" | {\n \t\tread tree prev_good_commit prev_good_job_number prev_good_job_id\n \n-\t\tif test \"$TRAVIS_JOB_ID\" = \"$prev_good_job_id\"\n+\t\tif test \"$CI_JOB_ID\" = \"$prev_good_job_id\"\n \t\tthen\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit has already been built and tested successfully by this build job.\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\telse\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.\n-\t\t\tThe log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id\n+\t\t\tThe log of that build job is available at $(url_for_job_id $prev_good_job_id)\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\tfi\n@@ -81,7 +102,6 @@ check_unignored_build_artifacts ()\n # and installing dependencies.\n set -ex\n \n-cache_dir=\"$HOME/travis-cache\"\n good_trees_file=\"$cache_dir/good-trees\"\n \n mkdir -p \"$cache_dir\"\n@@ -91,13 +111,11 @@ skip_good_tree\n \n if test -z \"$jobname\"\n then\n-\tjobname=\"$TRAVIS_OS_NAME-$CC\"\n+\tjobname=\"$CI_OS_NAME-$CC\"\n fi\n \n export DEVELOPER=1\n export DEFAULT_TEST_TARGET=prove\n-export GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n-export GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n export GIT_TEST_CLONE_2GB=YesPlease\n if [ \"$jobname\" = linux-gcc ]; then\n \texport CC=gcc-8\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 7aef39a2fd..d2045b63a6 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -69,7 +69,7 @@ do\n \tfi\n done\n \n-if [ $combined_trash_size -gt 0 ]\n+if [ -n \"$TRAVIS_JOB_ID\" -a $combined_trash_size -gt 0 ]\n then\n \techo \"------------------------------------------------------------------------\"\n \techo \"Trash directories embedded in this log can be extracted by running:\"\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex d3cdbac73f..7d0beb2832 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -5,6 +5,7 @@\n \n . ${0%/*}/lib.sh\n \n+test -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n gem install asciidoctor\n \n make check-builtins\n-- \ngitgitgadget\n\n"},{"id":"357261","messageId":"2b5d6785944ed1a66f0272896a45fece8521d10f.1536009027.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"[PATCH 4/9] tests: optionally write results as JUnit-style .xml","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-03T21:10:34Z","receivedAt":"2018-09-03T21:10:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis will come in handy when publishing the results of Git's test suite\nduring an automated VSTS CI run.\n\nSigned-off-by: Johannes Schindelin \n---\n t/.gitignore | 1 +\n t/test-lib.sh | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 99 insertions(+)\n\ndiff --git a/t/.gitignore b/t/.gitignore\nindex 348715f0e4..91cf5772fe 100644\n--- a/t/.gitignore\n+++ b/t/.gitignore\n@@ -2,3 +2,4 @@\n /test-results\n /.prove\n /chainlinttmp\n+/out/\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 8bb0f4348e..50a65a600e 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -288,6 +288,9 @@ do\n \t--verbose-log)\n \t\tverbose_log=t\n \t\tshift ;;\n+\t--write-junit-xml)\n+\t\twrite_junit_xml=t\n+\t\tshift ;;\n \t*)\n \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n \tesac\n@@ -431,11 +434,24 @@ trap 'exit $?' INT\n # the test_expect_* functions instead.\n \n test_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$*\"\n+\tfi\n \ttest_success=$(($test_success + 1))\n \tsay_color \"\" \"ok $test_count - $@\"\n }\n \n test_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_insert=\"\"\n+\t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n+\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\tjunit_insert=\"$junit_insert\"\n+\t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n+\tfi\n \ttest_failure=$(($test_failure + 1))\n \tsay_color error \"not ok $test_count - $1\"\n \tshift\n@@ -444,11 +460,19 @@ test_failure_ () {\n }\n \n test_known_broken_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (breakage fixed)\"\n+\tfi\n \ttest_fixed=$(($test_fixed+1))\n \tsay_color error \"ok $test_count - $@ # TODO known breakage vanished\"\n }\n \n test_known_broken_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (known breakage)\"\n+\tfi\n \ttest_broken=$(($test_broken+1))\n \tsay_color warn \"not ok $test_count - $@ # TODO known breakage\"\n }\n@@ -706,6 +730,10 @@ test_start_ () {\n \ttest_count=$(($test_count+1))\n \tmaybe_setup_verbose\n \tmaybe_setup_valgrind\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_start=$(test-tool date getnanos)\n+\tfi\n }\n \n test_finish_ () {\n@@ -743,6 +771,13 @@ test_skip () {\n \n \tcase \"$to_skip\" in\n \tt)\n+\t\tif test -n \"$write_junit_xml\"\n+\t\tthen\n+\t\t\tmessage=\"$(xml_attr_encode \"$skipped_reason\")\"\n+\t\t\twrite_junit_xml_testcase \"$1\" \\\n+\t\t\t\t\" \"\n+\t\tfi\n+\n \t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n@@ -758,9 +793,58 @@ test_at_end_hook_ () {\n \t:\n }\n \n+write_junit_xml () {\n+\tcase \"$1\" in\n+\t--truncate)\n+\t\t>\"$junit_xml_path\"\n+\t\tjunit_have_testcase=\n+\t\tshift\n+\t\t;;\n+\tesac\n+\tprintf '%s\\n' \"$@\" >>\"$junit_xml_path\"\n+}\n+\n+xml_attr_encode () {\n+\t# We do not translate CR to because BSD sed does not handle\n+\t# \\r in the regex. In practice, the output should not even have any\n+\t# carriage returns.\n+\tprintf '%s\\n' \"$@\" |\n+\tsed -e 's/&/\\&/g' -e \"s/'/\\'/g\" -e 's/\"/\\"/g' \\\n+\t\t-e 's//\\>/g' \\\n+\t\t-e 's/\t/\\ /g' -e 's/$/\\ /' -e '$s/ $//' |\n+\ttr -d '\\012\\015'\n+}\n+\n+write_junit_xml_testcase () {\n+\tjunit_attrs=\"name=\\\"$(xml_attr_encode \"$this_test.$test_count $1\")\\\"\"\n+\tshift\n+\tjunit_attrs=\"$junit_attrs classname=\\\"$this_test\\\"\"\n+\tjunit_attrs=\"$junit_attrs time=\\\"$(test-tool \\\n+\t\tdate getnanos $junit_start)\\\"\"\n+\twrite_junit_xml \"$(printf '%s\\n' \\\n+\t\t\" \" \"$@\" \" \")\"\n+\tjunit_have_testcase=t\n+}\n+\n test_done () {\n \tGIT_EXIT_OK=t\n \n+\tif test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n+\tthen\n+\t\ttest -n \"$junit_have_testcase\" || {\n+\t\t\tjunit_start=$(test-tool date getnanos)\n+\t\t\twrite_junit_xml_testcase \"all tests skipped\"\n+\t\t}\n+\n+\t\t# adjust the overall time\n+\t\tjunit_time=$(test-tool date getnanos $junit_suite_start)\n+\t\tsed \"s/]*/& time=\\\"$junit_time\\\"/\" \\\n+\t\t\t<\"$junit_xml_path\" >\"$junit_xml_path.new\"\n+\t\tmv \"$junit_xml_path.new\" \"$junit_xml_path\"\n+\n+\t\twrite_junit_xml \" \" \"\"\n+\tfi\n+\n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n \t\ttest_results_dir=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n@@ -996,6 +1080,7 @@ then\n else\n \tmkdir -p \"$TRASH_DIRECTORY\"\n fi\n+\n # Use -P to resolve symlinks in our working directory so that the cwd\n # in subprocesses like git equals our $PWD (for pathname comparisons).\n cd -P \"$TRASH_DIRECTORY\" || exit 1\n@@ -1009,6 +1094,19 @@ then\n \ttest_done\n fi\n \n+if test -n \"$write_junit_xml\"\n+then\n+\tjunit_xml_dir=\"$TEST_OUTPUT_DIRECTORY/out\"\n+\tmkdir -p \"$junit_xml_dir\"\n+\tjunit_xml_base=${0##*/}\n+\tjunit_xml_path=\"$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml\"\n+\tjunit_attrs=\"name=\\\"${junit_xml_base%.sh}\\\"\"\n+\tjunit_attrs=\"$junit_attrs timestamp=\\\"$(TZ=UTC \\\n+\t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n+\twrite_junit_xml --truncate \"\" \" \"\n+\tjunit_suite_start=$(test-tool date getnanos)\n+fi\n+\n # Provide an implementation of the 'yes' utility\n yes () {\n \tif test $# = 0\n-- \ngitgitgadget\n\n"},{"id":"357262","messageId":"127dfcfb09d3830fca92249cc56eaf68f5c55936.1536009027.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"[PATCH 5/9] ci/lib.sh: add support for VSTS CI","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-03T21:10:35Z","receivedAt":"2018-09-03T21:10:39Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis patch introduces a conditional arm that defines some environment\nvariables and a function that displays the URL given the job id (to\nidentify previous runs for known-good trees).\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 657ff88672..f2f6d70c72 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -19,6 +19,29 @@ then\n \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+elif test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n+then\n+\t# We are running in VSTS CI\n+\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n+\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n+\tCI_JOB_ID=\"$BUILD_BUILDID\"\n+\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n+\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n+\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n+\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n+\tCC=\"${CC:-gcc}\"\n+\n+\t# use a subdirectory of the cache dir (because the file share is shared\n+\t# among *all* phases)\n+\tcache_dir=\"$HOME/vsts-cache/$SYSTEM_PHASENAME\"\n+\n+\turl_for_job_id () {\n+\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n fi\n \n skip_branch_tip_with_tag () {\n-- \ngitgitgadget\n\n"},{"id":"357263","messageId":"ab66060c579aebebccf88a63559207d584a24d38.1536009027.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"[PATCH 6/9] Add a build definition for VSTS","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-03T21:10:37Z","receivedAt":"2018-09-03T21:10:40Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis commit adds a .vsts-ci.yml which is Visual Studio Team Services'\nequivalent to Travis CI's .travis.yml.\n\nTo make things a bit easier to understand, we refrain from using the\n`matrix` feature here because (while it is powerful) it can be a bit\nconfusing to users who are not familiar with CI setups. Therefore, we\nuse a separate phase even for similar configurations (such as GCC vs\nClang on Linux, GCC vs Clang on macOS).\n\nAlso, we make use of the shiny new feature we just introduced where the\ntest suite can output JUnit-style .xml files. This information is made\navailable in a nice UI that allows the viewer to filter by phase and/or\ntest number, and to see trends such as: number of (failing) tests, time\nspent running the test suite, etc.\n\nSigned-off-by: Johannes Schindelin \n---\n .vsts-ci.yml | 296 ++++++++++++++++++++++++++++++++++++++++++\n ci/mount-fileshare.sh | 26 ++++\n 2 files changed, 322 insertions(+)\n create mode 100644 .vsts-ci.yml\n create mode 100755 ci/mount-fileshare.sh\n\ndiff --git a/.vsts-ci.yml b/.vsts-ci.yml\nnew file mode 100644\nindex 0000000000..ff1d952df9\n--- /dev/null\n+++ b/.vsts-ci.yml\n@@ -0,0 +1,296 @@\n+resources:\n+- repo: self\n+ fetchDepth: 1\n+\n+phases:\n+- phase: linux_clang\n+ displayName: linux-clang\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n+\n+ sudo apt-get update\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin\n+\n+ export CC=clang\n+\n+ ci/install-dependencies.sh\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ sudo umount \"$HOME/vsts-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-clang'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: linux_gcc\n+ displayName: linux-gcc\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n+\n+ sudo apt-get update\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin\n+\n+ ci/install-dependencies.sh\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ sudo umount \"$HOME/vsts-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-gcc'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: osx_clang\n+ displayName: osx-clang\n+ condition: succeeded()\n+ queue:\n+ name: Hosted macOS\n+ steps:\n+ - bash: |\n+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n+\n+ export CC=clang\n+\n+ ci/install-dependencies.sh\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ umount \"$HOME/vsts-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-clang'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: osx_gcc\n+ displayName: osx-gcc\n+ condition: succeeded()\n+ queue:\n+ name: Hosted macOS\n+ steps:\n+ - bash: |\n+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n+\n+ ci/install-dependencies.sh\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ umount \"$HOME/vsts-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-gcc'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: gettext_poison\n+ displayName: GETTEXT_POISON\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n+\n+ sudo apt-get update\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev\n+\n+ export jobname=GETTEXT_POISON\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ sudo umount \"$HOME/vsts-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'gettext-poison'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: windows\n+ displayName: Windows\n+ condition: succeeded()\n+ queue:\n+ name: Hosted VS2017\n+ timeoutInMinutes: 240\n+ steps:\n+ - powershell: |\n+ # Helper to check the error level of the latest command (exit with error when appropriate)\n+ function c() { if (!$?) { exit(1) } }\n+\n+ net use s: \\\\gitfileshare.file.core.windows.net\\vsts-cache \"$(gitfileshare.pwd)\" /user:AZURE\\gitfileshare /persistent:no; c\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\vsts-cache\" S:\\; c\n+\n+ # Add build agent's MinGit to PATH\n+ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n+\n+ # Helper to initialize (or update) a Git worktree\n+ function init ($path, $url, $set_origin) {\n+ if (Test-Path $path) {\n+ cd $path; c\n+ if (Test-Path .git) {\n+ git init; c\n+ } else {\n+ git status\n+ }\n+ } else {\n+ git init $path; c\n+ cd $path; c\n+ }\n+ git config core.autocrlf false; c\n+ git config core.untrackedCache true; c\n+ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n+ git remote add origin $url; c\n+ }\n+ git fetch --depth=1 $url master; c\n+ git reset --hard FETCH_HEAD; c\n+ git clean -df; c\n+ }\n+\n+ # Initialize Git for Windows' SDK\n+ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n+ init \"$sdk_path\" \"https://git-for-windows.visualstudio.com/_git/git-sdk-64\" 0\n+ init usr\\src\\build-extra https://github.com/git-for-windows/build-extra 1\n+\n+ cd \"$(Build.SourcesDirectory)\"; c\n+\n+ $env:HOME = \"$(Build.SourcesDirectory)\"\n+ $env:MSYSTEM = \"MINGW64\"\n+ git-sdk-64\\git-cmd --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ . ci/lib.sh\n+\n+ make -j10 DEVELOPER=1 NO_PERL=1 || exit 1\n+ NO_PERL=1 NO_SVN_TESTS=1 GIT_TEST_OPTS=\\\"--quiet --write-junit-xml\\\" time make -j15 -k DEVELOPER=1 test || {\n+ NO_PERL=1 NO_SVN_TESTS=1 GIT_TEST_OPTS=\\\"-i -v -x\\\" make -k failed; exit 1\n+ }\n+\n+ save_good_tree\n+ \"@\n+\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\vsts-cache\"\n+ displayName: 'build & test'\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'windows'\n+ platform: Windows\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: linux32\n+ displayName: Linux32\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n+\n+ sudo apt-get update\n+ sudo apt-get -y install \\\n+ apt-transport-https \\\n+ ca-certificates \\\n+ curl \\\n+ software-properties-common\n+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n+ sudo add-apt-repository \\\n+ \"deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n+ $(lsb_release -cs) \\\n+ stable\"\n+ sudo apt-get update\n+ sudo apt-get -y install docker-ce\n+\n+ sudo AGENT_OS=\"$AGENT_OS\" BUILD_BUILDNUMBER=\"$BUILD_BUILDNUMBER\" BUILD_REPOSITORY_URI=\"$BUILD_REPOSITORY_URI\" BUILD_SOURCEBRANCH=\"$BUILD_SOURCEBRANCH\" BUILD_SOURCEVERSION=\"$BUILD_SOURCEVERSION\" SYSTEM_PHASENAME=\"$SYSTEM_PHASENAME\" SYSTEM_TASKDEFINITIONSURI=\"$SYSTEM_TASKDEFINITIONSURI\" SYSTEM_TEAMPROJECT=\"$SYSTEM_TEAMPROJECT\" CC=$CC MAKEFLAGS=-j3 bash -lxc ci/run-linux32-docker.sh || exit 1\n+\n+ sudo chmod a+r t/out/TEST-*.xml\n+\n+ sudo umount \"$HOME/vsts-cache\" || exit 1\n+ displayName: 'ci/run-linux32-docker.sh'\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux32'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: static_analysis\n+ displayName: StaticAnalysis\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n+\n+ sudo apt-get update\n+ sudo apt-get install -y coccinelle\n+\n+ export jobname=StaticAnalysis\n+\n+ ci/run-static-analysis.sh || exit 1\n+\n+ sudo umount \"$HOME/vsts-cache\" || exit 1\n+ displayName: 'ci/run-static-analysis.sh'\n+\n+- phase: documentation\n+ displayName: Documentation\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n+\n+ sudo apt-get update\n+ sudo apt-get install -y asciidoc xmlto asciidoctor\n+\n+ export ALREADY_HAVE_ASCIIDOCTOR=yes.\n+ export jobname=Documentation\n+\n+ ci/test-documentation.sh || exit 1\n+\n+ sudo umount \"$HOME/vsts-cache\" || exit 1\n+ displayName: 'ci/test-documentation.sh'\ndiff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\nnew file mode 100755\nindex 0000000000..5fb5f74b70\n--- /dev/null\n+++ b/ci/mount-fileshare.sh\n@@ -0,0 +1,26 @@\n+#!/bin/sh\n+\n+die () {\n+\techo \"$*\" >&2\n+\texit 1\n+}\n+\n+test $# = 4 ||\n+die \"Usage: $0 \n\nThe JUnit XML format lends itself to be presented in a powerful UI,\nwhere you can drill down to the information you are interested in very\nquickly.\n\nFor test failures, this usually means that you want to see the detailed\ntrace of the failing tests.\n\nWith Travis CI, we passed the `--verbose-log` option to get those\ntraces. However, that seems excessive, as we do not need/use the logs in\nalmost all of those cases: only when a test fails do we have a way to\ninclude the trace.\n\nSo let's do something different in VSTS: let's run all the tests with\n`--quiet` first, and only if a failure is encountered, try to trace the\ncommands as they are executed.\n\nOf course, we cannot turn on `--verbose-log` after the fact. So let's\njust re-run the test with all the same options, adding `--verbose-log`.\nAnd then munging the output file into the JUnit XML on the fly.\n\nNote: there is an off chance that re-running the test in verbose mode\n\"fixes\" the failures (and this does happen from time to time!). That is\na possibility we should be able to live with. Ideally, we would label\nthis as \"Passed upon rerun\", and that outcome even exists on VSTS, but\nit is not available when using the JUnit XML format for now:\nhttps://github.com/Microsoft/vsts-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n\nThis patch contains a slightly inelegant workaround for the p4 and\ngit-daemon tests: when we try to re-run the p4/git-daemon tests after\nthe daemon has been started already, we need to kill said daemon. We do\nthis by detecting the presence of the `kill_p4d` and `stop_git_daemon`\nshell functions and calling them if available.\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 40 +++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 39 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 50a65a600e..ea4ed250cc 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -85,6 +85,13 @@ done,*)\n \ttest \"$(cat \"$BASE.exit\")\" = 0\n \texit\n \t;;\n+*' --write-junit-xml '*)\n+\t# record how to call this script *with* --verbose-log, in case\n+\t# we encounter a breakage\n+\tjunit_rerun_options_sq=\"$(printf '%s\\n' \"$0\" --verbose-log -x \"$@\" |\n+\t\tsed -e \"s/'/'\\\\\\\\''/g\" -e \"s/^/'/\" -e \"s/\\$/'/\" |\n+\t\ttr '\\012' ' ')\"\n+\t;;\n esac\n \n # For repeatability, reset the environment to known value.\n@@ -445,10 +452,37 @@ test_ok_ () {\n test_failure_ () {\n \tif test -n \"$write_junit_xml\"\n \tthen\n+\t\tif test -z \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\tthen\n+\t\t\tcase \"$(type kill_p4d 2>/dev/null | head -n 1)\" in\n+\t\t\t*function*) kill_p4d;;\n+\t\t\tesac\n+\n+\t\t\tcase \"$(type stop_git_daemon 2>/dev/null |\n+\t\t\t\thead -n 1)\" in\n+\t\t\t*function*) stop_git_daemon;;\n+\t\t\tesac\n+\n+\t\t\t# re-run with --verbose-log\n+\t\t\techo \"# Re-running: $junit_rerun_options_sq\" >&2\n+\n+\t\t\tcd \"$TEST_DIRECTORY\" &&\n+\t\t\teval \"${TEST_SHELL_PATH}\" \"$junit_rerun_options_sq\" \\\n+\t\t\t\t>/dev/null 2>&1\n+\t\t\tstatus=$?\n+\n+\t\t\tsay_color \"\" \"$(test 0 = $status ||\n+\t\t\t\techo \"not \")ok $test_count - (re-ran with trace)\"\n+\t\t\tsay \"1..$test_count\"\n+\t\t\tGIT_EXIT_OK=t\n+\t\t\texit $status\n+\t\tfi\n+\n \t\tjunit_insert=\"\"\n \t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n-\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n+\t\t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n \t\tjunit_insert=\"$junit_insert\"\n \t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n \tfi\n@@ -733,6 +767,10 @@ test_start_ () {\n \tif test -n \"$write_junit_xml\"\n \tthen\n \t\tjunit_start=$(test-tool date getnanos)\n+\n+\t\t# truncate output\n+\t\ttest -z \"$GIT_TEST_TEE_OUTPUT_FILE\" ||\n+\t\t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n \tfi\n }\n \n-- \ngitgitgadget\n\n"},{"id":"357265","messageId":"dc1d890d718bd629dea03e6f4706e94282891313.1536009027.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"[PATCH 9/9] README: add a build badge (status of the VSTS build)","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-03T21:10:41Z","receivedAt":"2018-09-03T21:10:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nJust like so many other OSS projects, we now also have a build badge.\n\nSigned-off-by: Johannes Schindelin \n---\n README.md | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/README.md b/README.md\nindex f920a42fad..f260e78042 100644\n--- a/README.md\n+++ b/README.md\n@@ -1,3 +1,5 @@\n+[![Build Status](https://git.visualstudio.com/git/_apis/build/status/test-git.git)](https://git.visualstudio.com/git/_build/latest?definitionId=2)\n+\n Git - fast, scalable, distributed revision control system\n =========================================================\n \n-- \ngitgitgadget\n"},{"id":"357266","messageId":"3e83c6409068cf920d0c0a60e0e9a98b9bd6841a.1536009027.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"[PATCH 8/9] tests: record more stderr with --write-junit-xml in case of failure","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-09-03T21:10:40Z","receivedAt":"2018-09-03T21:10:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nSometimes, failures in a test case are actually caused by issues in\nearlier test cases.\n\nTo make it easier to see those issues, let's attach the output from\nbefore the failing test case (i.e. stdout/stderr since the previous\nfailing test case, or the start of the test script). This will be\nvisible in the \"Attachments\" of the details of the failed test.\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ea4ed250cc..6fc03d5a3b 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -484,6 +484,9 @@ test_failure_ () {\n \t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n \t\t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n \t\tjunit_insert=\"$junit_insert\"\n+\t\tjunit_insert=\"$junit_insert$(xml_attr_encode \\\n+\t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE.err\")\")\"\n+\t\t>\"$GIT_TEST_TEE_OUTPUT_FILE.err\"\n \t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n \tfi\n \ttest_failure=$(($test_failure + 1))\n@@ -768,9 +771,12 @@ test_start_ () {\n \tthen\n \t\tjunit_start=$(test-tool date getnanos)\n \n-\t\t# truncate output\n-\t\ttest -z \"$GIT_TEST_TEE_OUTPUT_FILE\" ||\n-\t\t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\t# append to future ; truncate output\n+\t\ttest -z \"$GIT_TEST_TEE_OUTPUT_FILE\" || {\n+\t\t\tcat \"$GIT_TEST_TEE_OUTPUT_FILE\" \\\n+\t\t\t\t>>\"$GIT_TEST_TEE_OUTPUT_FILE.err\"\n+\t\t\t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\t}\n \tfi\n }\n \n-- \ngitgitgadget\n\n"},{"id":"357280","messageId":"CAPig+cTYwnEt3_EoMZq7U265nEDRJc5uCtJbuVeBNTxqvWSPww@mail.gmail.com","threadId":"49270","inReplyTo":"18e6beec5f48ad5a17c7c7309e0507b01a53c2ec.1536009027.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/9] ci/lib.sh: encapsulate Travis-specific things","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-09-03T23:43:15Z","receivedAt":"2018-09-03T23:45:33Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget\n wrote:\n> The upcoming patches will allow building git.git via VSTS CI, where\n> variable names and URLs look a bit different than in Travis CI.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> @@ -28,7 +28,8 @@ osx-clang|osx-gcc)\n> # Uncomment this if you want to run perf tests:\n> # brew install gnu-time\n> - brew install git-lfs gettext\n> + test -z \"$BREW_INSTALL_PACKAGES\" ||\n> + eval brew install $BREW_INSTALL_PACKAGES\n\nThis 'eval' is unnecessary, isn't it?\n\n brew install $BREW_INSTALL_PACKAGES\n\nshould give the same result.\n\n> brew link --force gettext\n> brew install caskroom/cask/perforce\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> @@ -1,5 +1,26 @@\n> + BREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n"},{"id":"357283","messageId":"CAPig+cTJzguROwyGgyeUt5CrpKt2kyps04QD37wa_Bt3JPwWsg@mail.gmail.com","threadId":"49270","inReplyTo":"2b5d6785944ed1a66f0272896a45fece8521d10f.1536009027.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/9] tests: optionally write results as JUnit-style .xml","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-09-04T00:43:34Z","receivedAt":"2018-09-04T00:43:48Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget\n wrote:\n> This will come in handy when publishing the results of Git's test suite\n> during an automated VSTS CI run.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> @@ -431,11 +434,24 @@ trap 'exit $?' INT\n> test_failure_ () {\n> + if test -n \"$write_junit_xml\"\n> + then\n> + junit_insert=\" + junit_insert=\"$junit_insert $(xml_attr_encode \"$1\")\\\">\"\n> + junit_insert=\"$junit_insert $(xml_attr_encode \\\n> + \"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n> + junit_insert=\"$junit_insert\"\n\nThis is a genuine failure, so you're creating a node. Okay.\n\n> + write_junit_xml_testcase \"$1\" \" $junit_insert\"\n> + fi\n> @@ -444,11 +460,19 @@ test_failure_ () {\n> test_known_broken_ok_ () {\n> + if test -n \"$write_junit_xml\"\n> + then\n> + write_junit_xml_testcase \"$* (breakage fixed)\"\n> + fi\n> test_fixed=$(($test_fixed+1))\n> say_color error \"ok $test_count - $@ # TODO known breakage vanished\"\n> }\n\nThis was expected to fail but didn't, which means it probably needs\nsome sort of attention. test_known_broken_ok_() prints this result in\nthe 'error' color, and test_done() re-inforces that by printing a\nmessage, also in 'error' color:\n\n 42 known breakage(s) vanished; please update test(s)\n\nSo, should this emit a node also, perhaps with 'type'\nattribute set to \"warning\" or something? ()\n\n> @@ -758,9 +793,58 @@ test_at_end_hook_ () {\n> +xml_attr_encode () {\n> + # We do not translate CR to because BSD sed does not handle\n> + # \\r in the regex. In practice, the output should not even have any\n> + # carriage returns.\n> + printf '%s\\n' \"$@\" |\n> + sed -e 's/&/\\&/g' -e \"s/'/\\'/g\" -e 's/\"/\\"/g' \\\n> + -e 's//\\>/g' \\\n> + -e 's/ /\\ /g' -e 's/$/\\ /' -e '$s/ $//' |\n> + tr -d '\\012\\015'\n> +}\n\nIt's possible to insert a literal CR in the 'sed' expression, which\ndoes match correctly on BSD (and MacOS). For instance:\n\n CR=$(printf \"\\r\")\n sed -e \"s/$CR/\\ /g\"\n"},{"id":"357286","messageId":"CAPig+cRGHPfNqdgRr6H_D_siKCFZBDsUrcp0F+CWNCx884AJVg@mail.gmail.com","threadId":"49270","inReplyTo":"942bf423a461a3b44e2ff254c90907a539d7abbe.1536009027.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 7/9] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-09-04T04:30:45Z","receivedAt":"2018-09-04T04:31:00Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget\n wrote:\n> So let's do something different in VSTS: let's run all the tests with\n> `--quiet` first, and only if a failure is encountered, try to trace the\n> commands as they are executed. [...]\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> @@ -445,10 +452,37 @@ test_ok_ () {\n> test_failure_ () {\n> if test -n \"$write_junit_xml\"\n> then\n> + if test -z \"$GIT_TEST_TEE_OUTPUT_FILE\"\n> + then\n> + case \"$(type kill_p4d 2>/dev/null | head -n 1)\" in\n> + *function*) kill_p4d;;\n> + esac\n> +\n> + case \"$(type stop_git_daemon 2>/dev/null |\n> + head -n 1)\" in\n> + *function*) stop_git_daemon;;\n> + esac\n\nIn the long run, it might make more sense, and be more scalable, to\nhave those scripts define a \"prepare_for_rerun\" variable or function\nwhich this code then runs generically rather than having special\nknowledge of those facilities.\n\nI could imagine, for instance, test-lib.sh defining a no-op:\n\n test_failure_prepare_rerun () {}\n\nand then each of those scripts overriding the function:\n\n # in lib-git-p4.sh\n test_failure_prepare_rerun () {\n kill_p4d\n }\n\n # in lib-git-daemon.sh\n test_failure_prepare_rerun () {\n stop_git_daemon\n }\n\n> + # re-run with --verbose-log\n> + echo \"# Re-running: $junit_rerun_options_sq\" >&2\n> +\n> + cd \"$TEST_DIRECTORY\" &&\n> + eval \"${TEST_SHELL_PATH}\" \"$junit_rerun_options_sq\" \\\n> + >/dev/null 2>&1\n> + status=$?\n> +\n> + say_color \"\" \"$(test 0 = $status ||\n> + echo \"not \")ok $test_count - (re-ran with trace)\"\n> + say \"1..$test_count\"\n> + GIT_EXIT_OK=t\n> + exit $status\n> + fi\n> +\n> junit_insert=\" junit_insert=\"$junit_insert $(xml_attr_encode \"$1\")\\\">\"\n> junit_insert=\"$junit_insert $(xml_attr_encode \\\n> - \"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n> + \"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n> + >\"$GIT_TEST_TEE_OUTPUT_FILE\"\n> junit_insert=\"$junit_insert\"\n> write_junit_xml_testcase \"$1\" \" $junit_insert\"\n> fi\n\nThis junit-related stuff is getting pretty lengthy. I wonder if it\nwould make sense to pull it out to its own function at some point\n(again, in the long run).\n"},{"id":"357288","messageId":"nycvar.QRO.7.76.6.1809041256270.71@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAPig+cTJzguROwyGgyeUt5CrpKt2kyps04QD37wa_Bt3JPwWsg@mail.gmail.com","subject":"Re: [PATCH 4/9] tests: optionally write results as JUnit-style .xml","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-09-04T10:59:22Z","receivedAt":"2018-09-04T10:59:30Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Eric,\n\nOn Mon, 3 Sep 2018, Eric Sunshine wrote:\n\n> On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget\n> wrote:\n> > This will come in handy when publishing the results of Git's test suite\n> > during an automated VSTS CI run.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> > @@ -431,11 +434,24 @@ trap 'exit $?' INT\n> > test_failure_ () {\n> > + if test -n \"$write_junit_xml\"\n> > + then\n> > + junit_insert=\" > + junit_insert=\"$junit_insert $(xml_attr_encode \"$1\")\\\">\"\n> > + junit_insert=\"$junit_insert $(xml_attr_encode \\\n> > + \"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n> > + junit_insert=\"$junit_insert\"\n> \n> This is a genuine failure, so you're creating a node. Okay.\n> \n> > + write_junit_xml_testcase \"$1\" \" $junit_insert\"\n> > + fi\n> > @@ -444,11 +460,19 @@ test_failure_ () {\n> > test_known_broken_ok_ () {\n> > + if test -n \"$write_junit_xml\"\n> > + then\n> > + write_junit_xml_testcase \"$* (breakage fixed)\"\n> > + fi\n> > test_fixed=$(($test_fixed+1))\n> > say_color error \"ok $test_count - $@ # TODO known breakage vanished\"\n> > }\n> \n> This was expected to fail but didn't, which means it probably needs\n> some sort of attention. test_known_broken_ok_() prints this result in\n> the 'error' color, and test_done() re-inforces that by printing a\n> message, also in 'error' color:\n> \n> 42 known breakage(s) vanished; please update test(s)\n> \n> So, should this emit a node also, perhaps with 'type'\n> attribute set to \"warning\" or something? ( message=\"...\">)\n\nMy primary aim is to display the test results in the web interface, see\ne.g.\nhttps://git.visualstudio.com/git/_build/results?buildId=128&view=ms.vss-test-web.test-result-details\n\nThe parser for JUnit XML (and in fact, the JUnit XML schema itself) do not\nallow for such a warning. If you add a ``, then the build fails.\n\nAnd we do not want the build to fail. Historically, I saw quite a couple\nof \"vanished\" breakages depending on the platform where I ran the tests.\n\n> > @@ -758,9 +793,58 @@ test_at_end_hook_ () {\n> > +xml_attr_encode () {\n> > + # We do not translate CR to because BSD sed does not handle\n> > + # \\r in the regex. In practice, the output should not even have any\n> > + # carriage returns.\n> > + printf '%s\\n' \"$@\" |\n> > + sed -e 's/&/\\&/g' -e \"s/'/\\'/g\" -e 's/\"/\\"/g' \\\n> > + -e 's//\\>/g' \\\n> > + -e 's/ /\\ /g' -e 's/$/\\ /' -e '$s/ $//' |\n> > + tr -d '\\012\\015'\n> > +}\n> \n> It's possible to insert a literal CR in the 'sed' expression, which\n> does match correctly on BSD (and MacOS). For instance:\n> \n> CR=$(printf \"\\r\")\n> sed -e \"s/$CR/\\ /g\"\n\nOkay. But since we are talking about displaying some chunk of text, I\nwould rather just delete the CR here anyway.\n\nCiao,\nDscho\n"},{"id":"357289","messageId":"nycvar.QRO.7.76.6.1809041259480.71@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAPig+cTYwnEt3_EoMZq7U265nEDRJc5uCtJbuVeBNTxqvWSPww@mail.gmail.com","subject":"Re: [PATCH 2/9] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-09-04T11:04:31Z","receivedAt":"2018-09-04T11:04:39Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Eric,\n\nOn Mon, 3 Sep 2018, Eric Sunshine wrote:\n\n> On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget\n> wrote:\n> > The upcoming patches will allow building git.git via VSTS CI, where\n> > variable names and URLs look a bit different than in Travis CI.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> > @@ -28,7 +28,8 @@ osx-clang|osx-gcc)\n> > # Uncomment this if you want to run perf tests:\n> > # brew install gnu-time\n> > - brew install git-lfs gettext\n> > + test -z \"$BREW_INSTALL_PACKAGES\" ||\n> > + eval brew install $BREW_INSTALL_PACKAGES\n> \n> This 'eval' is unnecessary, isn't it?\n> \n> brew install $BREW_INSTALL_PACKAGES\n> \n> should give the same result.\n\nOh right! Fixed in https://github.com/gitgitgadget/git/pull/31 (and I also\nopened https://github.com/git/git/pull/531 to verify that Travis CI still\nworks).\n\nThanks,\nDscho\n\n> > brew link --force gettext\n> > brew install caskroom/cask/perforce\n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > @@ -1,5 +1,26 @@\n> > + BREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n> \n"},{"id":"357290","messageId":"nycvar.QRO.7.76.6.1809041304450.71@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAPig+cRGHPfNqdgRr6H_D_siKCFZBDsUrcp0F+CWNCx884AJVg@mail.gmail.com","subject":"Re: [PATCH 7/9] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-09-04T11:09:15Z","receivedAt":"2018-09-04T11:09:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Eric,\n\nOn Tue, 4 Sep 2018, Eric Sunshine wrote:\n\n> On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget\n> wrote:\n> > So let's do something different in VSTS: let's run all the tests with\n> > `--quiet` first, and only if a failure is encountered, try to trace the\n> > commands as they are executed. [...]\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> > @@ -445,10 +452,37 @@ test_ok_ () {\n> > test_failure_ () {\n> > if test -n \"$write_junit_xml\"\n> > then\n> > + if test -z \"$GIT_TEST_TEE_OUTPUT_FILE\"\n> > + then\n> > + case \"$(type kill_p4d 2>/dev/null | head -n 1)\" in\n> > + *function*) kill_p4d;;\n> > + esac\n> > +\n> > + case \"$(type stop_git_daemon 2>/dev/null |\n> > + head -n 1)\" in\n> > + *function*) stop_git_daemon;;\n> > + esac\n> \n> In the long run, it might make more sense, and be more scalable, to\n> have those scripts define a \"prepare_for_rerun\" variable or function\n> which this code then runs generically rather than having special\n> knowledge of those facilities.\n> \n> I could imagine, for instance, test-lib.sh defining a no-op:\n> \n> test_failure_prepare_rerun () {}\n> \n> and then each of those scripts overriding the function:\n> \n> # in lib-git-p4.sh\n> test_failure_prepare_rerun () {\n> kill_p4d\n> }\n> \n> # in lib-git-daemon.sh\n> test_failure_prepare_rerun () {\n> stop_git_daemon\n> }\n\nOr we could implement `test_atexit` (similar to `test_when_finished`, but\nto be executed at `test_done` time). I guess that's what the p4 and daemon\ntests really needed to begin with (and probably also the apache2-using\ntests).\n\n> \n> > + # re-run with --verbose-log\n> > + echo \"# Re-running: $junit_rerun_options_sq\" >&2\n> > +\n> > + cd \"$TEST_DIRECTORY\" &&\n> > + eval \"${TEST_SHELL_PATH}\" \"$junit_rerun_options_sq\" \\\n> > + >/dev/null 2>&1\n> > + status=$?\n> > +\n> > + say_color \"\" \"$(test 0 = $status ||\n> > + echo \"not \")ok $test_count - (re-ran with trace)\"\n> > + say \"1..$test_count\"\n> > + GIT_EXIT_OK=t\n> > + exit $status\n> > + fi\n> > +\n> > junit_insert=\" > junit_insert=\"$junit_insert $(xml_attr_encode \"$1\")\\\">\"\n> > junit_insert=\"$junit_insert $(xml_attr_encode \\\n> > - \"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n> > + \"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n> > + >\"$GIT_TEST_TEE_OUTPUT_FILE\"\n> > junit_insert=\"$junit_insert\"\n> > write_junit_xml_testcase \"$1\" \" $junit_insert\"\n> > fi\n> \n> This junit-related stuff is getting pretty lengthy. I wonder if it\n> would make sense to pull it out to its own function at some point\n> (again, in the long run).\n\nNow that you mention it... I agree. This is getting long.\n\nIn the short run, I have two things to consider, though: I want to make\nthis work first, then think about introducing a layer of abstraction, and\nI want to go on vacation tomorrow.\n\nSo I agree that this is something to be considered in the long run, i.e.\nnot right now ;-)\n\nThanks,\nDscho\n"},{"id":"357406","messageId":"CAE5ih7_GgkEaVXONy8ZY9j43PX6GD-mkD9vJ1t93upAfTkM8tw@mail.gmail.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1809041304450.71@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 7/9] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2018-09-05T05:32:57Z","receivedAt":"2018-09-05T05:33:01Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On 4 September 2018 at 12:09, Johannes Schindelin\n wrote:\n> Hi Eric,\n>\n> On Tue, 4 Sep 2018, Eric Sunshine wrote:\n>\n>> On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget\n>> wrote:\n>> > So let's do something different in VSTS: let's run all the tests with\n>> > `--quiet` first, and only if a failure is encountered, try to trace the\n>> > commands as they are executed. [...]\n\nIs this re-running just an individual test on its own or all the tests\nwithin a single file?\n\nThe latter shouldn't need this at all. And the former, I'm not sure\nwill actually work - most of the tests assume some particular p4\nstate. But perhaps I'm missing something?\n\nI also think it does look kind of ugly. And if there's one thing I've\nlearned, it's that the ugly hack you write today with the words \"we'll\ntidy this up later\" goes on to live with you forever!\n\n>> >\n>> > Signed-off-by: Johannes Schindelin \n>> > ---\n>> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n>> > @@ -445,10 +452,37 @@ test_ok_ () {\n>> > test_failure_ () {\n>> > if test -n \"$write_junit_xml\"\n>> > then\n>> > + if test -z \"$GIT_TEST_TEE_OUTPUT_FILE\"\n>> > + then\n>> > + case \"$(type kill_p4d 2>/dev/null | head -n 1)\" in\n>> > + *function*) kill_p4d;;\n>> > + esac\n>> > +\n>> > + case \"$(type stop_git_daemon 2>/dev/null |\n>> > + head -n 1)\" in\n>> > + *function*) stop_git_daemon;;\n>> > + esac\n>>\n>> In the long run, it might make more sense, and be more scalable, to\n>> have those scripts define a \"prepare_for_rerun\" variable or function\n>> which this code then runs generically rather than having special\n>> knowledge of those facilities.\n>>\n>> I could imagine, for instance, test-lib.sh defining a no-op:\n>>\n>> test_failure_prepare_rerun () {}\n>>\n>> and then each of those scripts overriding the function:\n>>\n>> # in lib-git-p4.sh\n>> test_failure_prepare_rerun () {\n>> kill_p4d\n>> }\n>>\n>> # in lib-git-daemon.sh\n>> test_failure_prepare_rerun () {\n>> stop_git_daemon\n>> }\n>\n> Or we could implement `test_atexit` (similar to `test_when_finished`, but\n> to be executed at `test_done` time). I guess that's what the p4 and daemon\n> tests really needed to begin with (and probably also the apache2-using\n> tests).\n>\n>>\n>> > + # re-run with --verbose-log\n>> > + echo \"# Re-running: $junit_rerun_options_sq\" >&2\n>> > +\n>> > + cd \"$TEST_DIRECTORY\" &&\n>> > + eval \"${TEST_SHELL_PATH}\" \"$junit_rerun_options_sq\" \\\n>> > + >/dev/null 2>&1\n>> > + status=$?\n>> > +\n>> > + say_color \"\" \"$(test 0 = $status ||\n>> > + echo \"not \")ok $test_count - (re-ran with trace)\"\n>> > + say \"1..$test_count\"\n>> > + GIT_EXIT_OK=t\n>> > + exit $status\n>> > + fi\n>> > +\n>> > junit_insert=\"> > junit_insert=\"$junit_insert $(xml_attr_encode \"$1\")\\\">\"\n>> > junit_insert=\"$junit_insert $(xml_attr_encode \\\n>> > - \"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n>> > + \"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n>> > + >\"$GIT_TEST_TEE_OUTPUT_FILE\"\n>> > junit_insert=\"$junit_insert\"\n>> > write_junit_xml_testcase \"$1\" \" $junit_insert\"\n>> > fi\n>>\n>> This junit-related stuff is getting pretty lengthy. I wonder if it\n>> would make sense to pull it out to its own function at some point\n>> (again, in the long run).\n>\n> Now that you mention it... I agree. This is getting long.\n>\n> In the short run, I have two things to consider, though: I want to make\n> this work first, then think about introducing a layer of abstraction, and\n> I want to go on vacation tomorrow.\n>\n> So I agree that this is something to be considered in the long run, i.e.\n> not right now ;-)\n>\n> Thanks,\n> Dscho\n"},{"id":"357418","messageId":"nycvar.QRO.7.76.6.1809051430390.71@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAE5ih7_GgkEaVXONy8ZY9j43PX6GD-mkD9vJ1t93upAfTkM8tw@mail.gmail.com","subject":"Re: [PATCH 7/9] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-09-05T12:39:10Z","receivedAt":"2018-09-05T12:39:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Luke,\n\nOn Wed, 5 Sep 2018, Luke Diamand wrote:\n\n> On 4 September 2018 at 12:09, Johannes Schindelin\n> wrote:\n> >\n> > On Tue, 4 Sep 2018, Eric Sunshine wrote:\n> >\n> >> On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget\n> >> wrote:\n> >> > So let's do something different in VSTS: let's run all the tests with\n> >> > `--quiet` first, and only if a failure is encountered, try to trace the\n> >> > commands as they are executed. [...]\n> \n> Is this re-running just an individual test on its own or all the tests\n> within a single file?\n\nUpon encountering a failed test, it is re-running the entire test script\nafresh.\n\n> The latter shouldn't need this at all.\n\nPlease do not let me die dumb. In other words, I would love for you to\nexplain what exactly you mean by that sentence.\n\n> And the former, I'm not sure will actually work - most of the tests\n> assume some particular p4 state. But perhaps I'm missing something?\n\nNo, the former would not work at all. Not only for the p4 tests: Git's\ntests frequently commit the deadly sin of relying on output of one another\n(wreaking havoc e.g. when test cases are skipped due to missing\nprerequisites, and latter test cases relying on their output). It is not\nthe only thing that is wrong with the test suite, of course.\n\n> I also think it does look kind of ugly. And if there's one thing I've\n> learned, it's that the ugly hack you write today with the words \"we'll\n> tidy this up later\" goes on to live with you forever!\n\nOkay.\n\n(And having read lib-git-p4.sh, I kind of see where you learned that.)\n\nBut maybe you also have some splendid idea what to do instead? Because\ndoing something about it, that we need. We can't just say \"oh, the only\nsolution we found is ugly, so let's not do it at all\".\n\nI am even going so far as to say: unless you have a better idea, it is\npretty detrimental to criticize the current approach. It is the opposite\nof constructive.\n\nSo let's hear some ideas how to improve the situation, m'kay?\n\nJust as a reminder, this is the problem I want to solve: I want to run the\ntests in a light-weight manner, with minimal output, and only in case of\nan error do I want to crank up the verbosity. Instead of wasting most of the\neffort to log everything and then throwing it away in most of the common\ncases, I suggest to re-run the entire test.\n\nWhat do you suggest to solve this?\n\nCiao,\nJohannes\n\n> >> > Signed-off-by: Johannes Schindelin \n> >> > ---\n> >> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> >> > @@ -445,10 +452,37 @@ test_ok_ () {\n> >> > test_failure_ () {\n> >> > if test -n \"$write_junit_xml\"\n> >> > then\n> >> > + if test -z \"$GIT_TEST_TEE_OUTPUT_FILE\"\n> >> > + then\n> >> > + case \"$(type kill_p4d 2>/dev/null | head -n 1)\" in\n> >> > + *function*) kill_p4d;;\n> >> > + esac\n> >> > +\n> >> > + case \"$(type stop_git_daemon 2>/dev/null |\n> >> > + head -n 1)\" in\n> >> > + *function*) stop_git_daemon;;\n> >> > + esac\n> >>\n> >> In the long run, it might make more sense, and be more scalable, to\n> >> have those scripts define a \"prepare_for_rerun\" variable or function\n> >> which this code then runs generically rather than having special\n> >> knowledge of those facilities.\n> >>\n> >> I could imagine, for instance, test-lib.sh defining a no-op:\n> >>\n> >> test_failure_prepare_rerun () {}\n> >>\n> >> and then each of those scripts overriding the function:\n> >>\n> >> # in lib-git-p4.sh\n> >> test_failure_prepare_rerun () {\n> >> kill_p4d\n> >> }\n> >>\n> >> # in lib-git-daemon.sh\n> >> test_failure_prepare_rerun () {\n> >> stop_git_daemon\n> >> }\n> >\n> > Or we could implement `test_atexit` (similar to `test_when_finished`, but\n> > to be executed at `test_done` time). I guess that's what the p4 and daemon\n> > tests really needed to begin with (and probably also the apache2-using\n> > tests).\n> >\n> >>\n> >> > + # re-run with --verbose-log\n> >> > + echo \"# Re-running: $junit_rerun_options_sq\" >&2\n> >> > +\n> >> > + cd \"$TEST_DIRECTORY\" &&\n> >> > + eval \"${TEST_SHELL_PATH}\" \"$junit_rerun_options_sq\" \\\n> >> > + >/dev/null 2>&1\n> >> > + status=$?\n> >> > +\n> >> > + say_color \"\" \"$(test 0 = $status ||\n> >> > + echo \"not \")ok $test_count - (re-ran with trace)\"\n> >> > + say \"1..$test_count\"\n> >> > + GIT_EXIT_OK=t\n> >> > + exit $status\n> >> > + fi\n> >> > +\n> >> > junit_insert=\" >> > junit_insert=\"$junit_insert $(xml_attr_encode \"$1\")\\\">\"\n> >> > junit_insert=\"$junit_insert $(xml_attr_encode \\\n> >> > - \"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n> >> > + \"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n> >> > + >\"$GIT_TEST_TEE_OUTPUT_FILE\"\n> >> > junit_insert=\"$junit_insert\"\n> >> > write_junit_xml_testcase \"$1\" \" $junit_insert\"\n> >> > fi\n> >>\n> >> This junit-related stuff is getting pretty lengthy. I wonder if it\n> >> would make sense to pull it out to its own function at some point\n> >> (again, in the long run).\n> >\n> > Now that you mention it... I agree. This is getting long.\n> >\n> > In the short run, I have two things to consider, though: I want to make\n> > this work first, then think about introducing a layer of abstraction, and\n> > I want to go on vacation tomorrow.\n> >\n> > So I agree that this is something to be considered in the long run, i.e.\n> > not right now ;-)\n> >\n> > Thanks,\n> > Dscho\n> \n"},{"id":"357419","messageId":"CAE5ih787aEOAGoC9Ze8qgq+veLQYNkpKgjAG7MpkP0=nyQ5MMA@mail.gmail.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1809051430390.71@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 7/9] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2018-09-05T13:03:14Z","receivedAt":"2018-09-05T13:03:17Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On 5 September 2018 at 13:39, Johannes Schindelin\n wrote:\n> Hi Luke,\n>\n> On Wed, 5 Sep 2018, Luke Diamand wrote:\n>\n>> On 4 September 2018 at 12:09, Johannes Schindelin\n>> wrote:\n>> >\n>> > On Tue, 4 Sep 2018, Eric Sunshine wrote:\n>> >\n>> >> On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget\n>> >> wrote:\n>> >> > So let's do something different in VSTS: let's run all the tests with\n>> >> > `--quiet` first, and only if a failure is encountered, try to trace the\n>> >> > commands as they are executed. [...]\n>>\n>> Is this re-running just an individual test on its own or all the tests\n>> within a single file?\n>\n> Upon encountering a failed test, it is re-running the entire test script\n> afresh.\n>\n>> The latter shouldn't need this at all.\n>\n> Please do not let me die dumb. In other words, I would love for you to\n> explain what exactly you mean by that sentence.\n\nJust re-run the script. You shouldn't need to kill p4d, as each script\nstarts up its own instance of p4d, and shuts it down when it exits.\n\n$ cd t\n$ ./t9800-git-p4-basic.sh\nCtrl^C\n$ ./t9800-git-p4-basic.sh -v\n\nThere's a cleanup() function in lib-git-p4.sh which kills the p4d\nserver, and that's invoked via:\n\n trap cleanup EXIT\n\nThat's the only cleanup that each of the scripts require AFAIK.\n\n>\n>> And the former, I'm not sure will actually work - most of the tests\n>> assume some particular p4 state. But perhaps I'm missing something?\n>\n> No, the former would not work at all. Not only for the p4 tests: Git's\n> tests frequently commit the deadly sin of relying on output of one another\n> (wreaking havoc e.g. when test cases are skipped due to missing\n> prerequisites, and latter test cases relying on their output). It is not\n> the only thing that is wrong with the test suite, of course.\n>\n>> I also think it does look kind of ugly. And if there's one thing I've\n>> learned, it's that the ugly hack you write today with the words \"we'll\n>> tidy this up later\" goes on to live with you forever!\n>\n> Okay.\n>\n> (And having read lib-git-p4.sh, I kind of see where you learned that.)\n>\n> But maybe you also have some splendid idea what to do instead? Because\n> doing something about it, that we need. We can't just say \"oh, the only\n> solution we found is ugly, so let's not do it at all\".\n>\n> I am even going so far as to say: unless you have a better idea, it is\n> pretty detrimental to criticize the current approach. It is the opposite\n> of constructive.\n>\n> So let's hear some ideas how to improve the situation, m'kay?\n>\n> Just as a reminder, this is the problem I want to solve: I want to run the\n> tests in a light-weight manner, with minimal output, and only in case of\n> an error do I want to crank up the verbosity. Instead of wasting most of the\n> effort to log everything and then throwing it away in most of the common\n> cases, I suggest to re-run the entire test.\n>\n> What do you suggest to solve this?\n>\n\nI don't know about any other tests, but the git-p4 tests don't take\nany longer in verbose mode. So one simple solution is to just run it\nin verbose mode - unless there are other tests which have more\noverhead.\n\nThe trap/exit/cleanup method that the git-p4 tests already use would\nseem to be ideally suited to cleaning up everything on exit.\n\nThere might be some specific tests where this doesn't quite work out,\nif you let me know what they are I can have a look at fixing them for\nyou.\n\nThanks,\nLuke\n"},{"id":"357463","messageId":"CAPig+cQkC4-6DyQdJHc8QUfBx+L6o1SK4ODy4MoP44+VHHk96Q@mail.gmail.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1809051430390.71@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 7/9] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-09-05T18:38:34Z","receivedAt":"2018-09-05T18:38:48Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Wed, Sep 5, 2018 at 8:39 AM Johannes Schindelin\n wrote:\n> So let's hear some ideas how to improve the situation, m'kay?\n> Just as a reminder, this is the problem I want to solve: I want to run the\n> tests in a light-weight manner, with minimal output, and only in case of\n> an error do I want to crank up the verbosity. Instead of wasting most of the\n> effort to log everything and then throwing it away in most of the common\n> cases, I suggest to re-run the entire test.\n\nWhat about the very different approach of capturing the full \"verbose\"\noutput the executed tests in addition to whatever is actually output\nto the terminal? If a test fails, then (and only then) you can insert\nthe captured verbose output into the JUnit XML file. This way (if we\nalways have the full verbose output at hand), you don't need to re-run\nthe test at all.\n\nI've cc:'d Peff and Jonathan since I believe they are more familiar\nwith how all the capturing / output-redirection works in the test\nsuite.\n"},{"id":"357466","messageId":"179314eb-774a-bc98-4a41-f79ec78c8466@gmail.com","threadId":"49270","inReplyTo":"18e6beec5f48ad5a17c7c7309e0507b01a53c2ec.1536009027.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/9] ci/lib.sh: encapsulate Travis-specific things","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2018-09-05T18:57:44Z","receivedAt":"2018-09-05T18:57:56Z","isPatch":true,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"On 9/3/2018 11:10 PM, Johannes Schindelin via GitGitGadget wrote:\n\n> +if test -n \"$TRAVIS_COMMIT\"\n> +then\n> +\t# We are running within Travis CI\n\nPersonally, I'd find a check like\n\nif test \"$TRAVIS\" = \"true\"\n\nmore speaking (also see [1]).\n\n[1] https://docs.travis-ci.com/user/environment-variables/\n\n-- \nSebastian Schuberth\n\n"},{"id":"357468","messageId":"25914342-ff35-90b2-860e-6aff5c109f14@gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Sebastian Schuberth","fromEmail":"sschuberth@gmail.com","sentAt":"2018-09-05T19:01:36Z","receivedAt":"2018-09-05T19:02:10Z","isPatch":true,"sender":{"key":"sschuberth@gmail.com","avatar":"https://avatars.githubusercontent.com/u/349154?v=4"},"body":"On 9/3/2018 11:10 PM, Johannes Schindelin via GitGitGadget wrote:\n\n> The one sad part about this is the Windows support. Travis lacks it, and we\n> work around that by using Visual Studio Team Services (VSTS) indirectly: one\n> phase in Travis would trigger a build, wait for its log, and then paste that\n> log.\n\nI'm sorry if this has been discussed before, but as this recap doesn't \nmention it: Has AppVeyor been considered as an option? It seems to be \nthe defacto standard for Windows CI for projects on GitHub.\n\n-- \nSebastian Schuberth\n"},{"id":"357470","messageId":"CAGZ79kZiA+0R6QB5qgVDUSZBZp_Nv+Yi-SQC5yHeWTzJnw1y8w@mail.gmail.com","threadId":"49270","inReplyTo":"25914342-ff35-90b2-860e-6aff5c109f14@gmail.com","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-09-05T19:08:30Z","receivedAt":"2018-09-05T19:08:44Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Sep 5, 2018 at 12:02 PM Sebastian Schuberth\n wrote:\n>\n> On 9/3/2018 11:10 PM, Johannes Schindelin via GitGitGadget wrote:\n>\n> > The one sad part about this is the Windows support. Travis lacks it, and we\n> > work around that by using Visual Studio Team Services (VSTS) indirectly: one\n> > phase in Travis would trigger a build, wait for its log, and then paste that\n> > log.\n>\n> I'm sorry if this has been discussed before, but as this recap doesn't\n> mention it: Has AppVeyor been considered as an option? It seems to be\n> the defacto standard for Windows CI for projects on GitHub.\n\nThere was\nhttps://public-inbox.org/git/BAY169-W30CD27F2F7606F4DF52944A78F0@phx.gbl/\nand\nhttps://public-inbox.org/git/alpine.DEB.2.20.1703241242210.17768@tvnag.unkk.fr/\n"},{"id":"357478","messageId":"20180905202445.GA8953@sigill.intra.peff.net","threadId":"49270","inReplyTo":"CAPig+cQkC4-6DyQdJHc8QUfBx+L6o1SK4ODy4MoP44+VHHk96Q@mail.gmail.com","subject":"Re: [PATCH 7/9] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-09-05T20:24:46Z","receivedAt":"2018-09-05T20:24:49Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 05, 2018 at 02:38:34PM -0400, Eric Sunshine wrote:\n\n> On Wed, Sep 5, 2018 at 8:39 AM Johannes Schindelin\n> wrote:\n> > So let's hear some ideas how to improve the situation, m'kay?\n> > Just as a reminder, this is the problem I want to solve: I want to run the\n> > tests in a light-weight manner, with minimal output, and only in case of\n> > an error do I want to crank up the verbosity. Instead of wasting most of the\n> > effort to log everything and then throwing it away in most of the common\n> > cases, I suggest to re-run the entire test.\n> \n> What about the very different approach of capturing the full \"verbose\"\n> output the executed tests in addition to whatever is actually output\n> to the terminal? If a test fails, then (and only then) you can insert\n> the captured verbose output into the JUnit XML file. This way (if we\n> always have the full verbose output at hand), you don't need to re-run\n> the test at all.\n> \n> I've cc:'d Peff and Jonathan since I believe they are more familiar\n> with how all the capturing / output-redirection works in the test\n> suite.\n\nI don't think there's much to know beyond what you wrote. The\n\"--verbose\" case does not really cost any more than the non-verbose one,\nbecause the only difference is whether output goes to a file versus\n/dev/null. But the commands all call write() regardless.\n\nFor --verbose-log, it does of course cost a little extra to run one\n`tee` per script, and to write a few kb of logs. I doubt those are\nmeasurable versus the rest of a script run, but I'm happy to be\ndisproven by numbers. There are some gymnastics done to re-exec the test\nscript with the same shell, but AFAIK those are robust and don't cost a\nlot (again, one extra process per script run).\n\nI'm not overly concerned about the cost of re-running a test, since the\nidea is that failed tests should be rare. I would be a little worried\nabout flaky tests mysteriously righting themselves on the second run (so\nyou know a failure happened, but you have no good output to describe\nit).\n\nI do agree that a test_atexit() cleanup would make a lot more sense than\nwhat's in the original patch. And that's nicer than the exit trap we're\nusing already, because you may note that each caller has to manually\nrestore the original 'die' handler that test-lib.sh installs.\n\nThat would also help with bitrot. If this funky cleanup case only causes\nproblems with junit output, then other people are likely to forget to do\nit, and the mess falls onto the junit folks (i.e., Dscho). But if the\ntests start _relying_ on test_atexit() being called (i.e., don't call\nstop_git_daemon manually, but assume that the atexit handler does so),\nthen the responsible authors are a lot more likely to notice and fix it\nearly.\n\n-Peff\n"},{"id":"358146","messageId":"nycvar.QRO.7.76.6.1809141338360.73@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAE5ih787aEOAGoC9Ze8qgq+veLQYNkpKgjAG7MpkP0=nyQ5MMA@mail.gmail.com","subject":"Re: [PATCH 7/9] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-09-14T18:46:09Z","receivedAt":"2018-09-14T18:46:21Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Luke,\n\nOn Wed, 5 Sep 2018, Luke Diamand wrote:\n\n> On 5 September 2018 at 13:39, Johannes Schindelin\n> wrote:\n> >\n> > On Wed, 5 Sep 2018, Luke Diamand wrote:\n> >\n> >> On 4 September 2018 at 12:09, Johannes Schindelin\n> >> wrote:\n> >> >\n> >> > On Tue, 4 Sep 2018, Eric Sunshine wrote:\n> >> >\n> >> >> On Mon, Sep 3, 2018 at 5:10 PM Johannes Schindelin via GitGitGadget\n> >> >> wrote:\n> >> >> > So let's do something different in VSTS: let's run all the tests with\n> >> >> > `--quiet` first, and only if a failure is encountered, try to trace the\n> >> >> > commands as they are executed. [...]\n> >>\n> >> Is this re-running just an individual test on its own or all the tests\n> >> within a single file?\n> >\n> > Upon encountering a failed test, it is re-running the entire test script\n> > afresh.\n> >\n> >> The latter shouldn't need this at all.\n> >\n> > Please do not let me die dumb. In other words, I would love for you to\n> > explain what exactly you mean by that sentence.\n> \n> Just re-run the script. You shouldn't need to kill p4d, as each script\n> starts up its own instance of p4d, and shuts it down when it exits.\n> \n> $ cd t\n> $ ./t9800-git-p4-basic.sh\n> Ctrl^C\n\nThat Ctrl^C is not possible in the automated builds (which are the subject\nof this patch series).\n\nWhat *is* possible is to detect in the test script that there was a\nbreakage, and then re-run the test script.\n\nUnfortunately, at this stage the first test script has not exited yet, so\nyour p4d has not exited yet, and we have to clean up in some way before\nre-running (because the port in use is determined by the test script's\nnumber, so we really need to kill the old daemon).\n\nAnd this is exactly what my patches try to achieve here.\n\n> $ ./t9800-git-p4-basic.sh -v\n> \n> There's a cleanup() function in lib-git-p4.sh which kills the p4d\n> server, and that's invoked via:\n> \n> trap cleanup EXIT\n> \n> That's the only cleanup that each of the scripts require AFAIK.\n\nGood.\n\nHowever, we really will want to consider introducing something consistent\nhere, something that works for *all* test scripts. IOW if any test script\nwants to start a process for the life-time of that script, and needs to\nkill that process at the end, we will want to have a consistent,\ndocumented way to register a function (or commands) to be called at the\nend.\n\nAnd that function (or commands) need to be run also when stopping\neverything in preparation for a re-run.\n\n> >> And the former, I'm not sure will actually work - most of the tests\n> >> assume some particular p4 state. But perhaps I'm missing something?\n> >\n> > No, the former would not work at all. Not only for the p4 tests: Git's\n> > tests frequently commit the deadly sin of relying on output of one another\n> > (wreaking havoc e.g. when test cases are skipped due to missing\n> > prerequisites, and latter test cases relying on their output). It is not\n> > the only thing that is wrong with the test suite, of course.\n> >\n> >> I also think it does look kind of ugly. And if there's one thing I've\n> >> learned, it's that the ugly hack you write today with the words \"we'll\n> >> tidy this up later\" goes on to live with you forever!\n> >\n> > Okay.\n> >\n> > (And having read lib-git-p4.sh, I kind of see where you learned that.)\n> >\n> > But maybe you also have some splendid idea what to do instead? Because\n> > doing something about it, that we need. We can't just say \"oh, the only\n> > solution we found is ugly, so let's not do it at all\".\n> >\n> > I am even going so far as to say: unless you have a better idea, it is\n> > pretty detrimental to criticize the current approach. It is the opposite\n> > of constructive.\n> >\n> > So let's hear some ideas how to improve the situation, m'kay?\n> >\n> > Just as a reminder, this is the problem I want to solve: I want to run the\n> > tests in a light-weight manner, with minimal output, and only in case of\n> > an error do I want to crank up the verbosity. Instead of wasting most of the\n> > effort to log everything and then throwing it away in most of the common\n> > cases, I suggest to re-run the entire test.\n> >\n> > What do you suggest to solve this?\n> >\n> \n> I don't know about any other tests, but the git-p4 tests don't take\n> any longer in verbose mode.\n\nThat statement is too general to be correct. It may be the case in your\nsetup, but I found that even a redirected, chatty stderr can slow down\nthings on Windows substantially, probably due to the POSIX emulation going\non in the background that needs to prepare for all kinds of eventualities\n(that we do not even need, but the MSYS2 runtime has no way of knowing\nthat).\n\n> So one simple solution is to just run it in verbose mode - unless there\n> are other tests which have more overhead.\n\nIt is a bit of an overhead. As the entire test run on Windows takes\n*quite* a while (due to said POSIX emulation issues), I was not able to\nquantify it exactly.\n\n> The trap/exit/cleanup method that the git-p4 tests already use would\n> seem to be ideally suited to cleaning up everything on exit.\n\nYes. On exit. But I cannot exit here, not before re-running the script.\n\n> There might be some specific tests where this doesn't quite work out,\n> if you let me know what they are I can have a look at fixing them for\n> you.\n\nThanks. I will let you know when I encounter other tests than the p4 and\ngit-daemon tests (which I fixed in this patch series).\n\nCiao,\nDscho\n"},{"id":"358147","messageId":"nycvar.QRO.7.76.6.1809141346340.73@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAPig+cQkC4-6DyQdJHc8QUfBx+L6o1SK4ODy4MoP44+VHHk96Q@mail.gmail.com","subject":"Re: [PATCH 7/9] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-09-14T18:51:36Z","receivedAt":"2018-09-14T18:51:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Eric,\n\nOn Wed, 5 Sep 2018, Eric Sunshine wrote:\n\n> On Wed, Sep 5, 2018 at 8:39 AM Johannes Schindelin\n> wrote:\n> > So let's hear some ideas how to improve the situation, m'kay? Just as\n> > a reminder, this is the problem I want to solve: I want to run the\n> > tests in a light-weight manner, with minimal output, and only in case\n> > of an error do I want to crank up the verbosity. Instead of wasting\n> > most of the effort to log everything and then throwing it away in most\n> > of the common cases, I suggest to re-run the entire test.\n> \n> What about the very different approach of capturing the full \"verbose\"\n> output the executed tests in addition to whatever is actually output\n> to the terminal?\n\nI fear it is not really possible to do a \"verbose but not really\" mode. I\nwant the console output to be quiet, there is no use in chatting up the\nbuild log with these messages, as we have to run the tests in parallel, so\nthe output would be utterly hard to interpret anyway. At the same time, I\nwant verbose output for use in the test results. It is not really possible\nto `tee` all output, then \"quiet down\" the part that makes it into the\nlog.\n\n> If a test fails, then (and only then) you can insert the captured\n> verbose output into the JUnit XML file.\n\nYep. That's what my patch series does.\n\nIf a test fails, then (and only then) I re-run the script with verbose\noutput that is immediately moved into that JUnit XML file.\n\n> This way (if we always have the full verbose output at hand), you don't\n> need to re-run the test at all.\n\nBut that way, if we always have the full verbose output, the build log\nwill be unnecessarily verbose and confusing.\n\n> I've cc:'d Peff and Jonathan since I believe they are more familiar\n> with how all the capturing / output-redirection works in the test\n> suite.\n\nOkay.\n\nCiao,\nDscho\n\nP.S.: An unintended side effect of re-running the tests is to identify\nflakey tests. I do not yet have a way to represent this outcome in the\ntest result, but I deem this an additional benefit in favor of keeping my\ncurrent strategy.\n"},{"id":"358149","messageId":"nycvar.QRO.7.76.6.1809141351590.73@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"20180905202445.GA8953@sigill.intra.peff.net","subject":"Re: [PATCH 7/9] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-09-14T19:04:17Z","receivedAt":"2018-09-14T19:04:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Peff,\n\nOn Wed, 5 Sep 2018, Jeff King wrote:\n\n> On Wed, Sep 05, 2018 at 02:38:34PM -0400, Eric Sunshine wrote:\n> \n> > On Wed, Sep 5, 2018 at 8:39 AM Johannes Schindelin\n> > wrote:\n> > > So let's hear some ideas how to improve the situation, m'kay? Just\n> > > as a reminder, this is the problem I want to solve: I want to run\n> > > the tests in a light-weight manner, with minimal output, and only in\n> > > case of an error do I want to crank up the verbosity. Instead of\n> > > wasting most of the effort to log everything and then throwing it\n> > > away in most of the common cases, I suggest to re-run the entire\n> > > test.\n> > \n> > What about the very different approach of capturing the full \"verbose\"\n> > output the executed tests in addition to whatever is actually output\n> > to the terminal? If a test fails, then (and only then) you can insert\n> > the captured verbose output into the JUnit XML file. This way (if we\n> > always have the full verbose output at hand), you don't need to re-run\n> > the test at all.\n> > \n> > I've cc:'d Peff and Jonathan since I believe they are more familiar\n> > with how all the capturing / output-redirection works in the test\n> > suite.\n> \n> I don't think there's much to know beyond what you wrote. The\n> \"--verbose\" case does not really cost any more than the non-verbose one,\n> because the only difference is whether output goes to a file versus\n> /dev/null. But the commands all call write() regardless.\n\nThis really only holds true on systems where redirected output is\nessentially a no-op.\n\nFor the MSYS2-emulated case, this is incorrect. And as that case is\nalready suffering, time-wise, I would be loathe to pile more onto it.\n\n> For --verbose-log, it does of course cost a little extra to run one\n> `tee` per script, and to write a few kb of logs. I doubt those are\n> measurable versus the rest of a script run, but I'm happy to be\n> disproven by numbers.\n\nI tried, and failed, to quantify this properly. Essentially because a\nsingle test run takes over 1h15m, so I tried to do it in the cloud, but\nthose timings are too variable for anything approaching an accurate\nmeasurement.\n\n> There are some gymnastics done to re-exec the test script with the same\n> shell, but AFAIK those are robust and don't cost a lot (again, one extra\n> process per script run).\n\nIt *is* actually a bit worse here (but only in the case of failures, which\nshould be the minority of the cases): imagine an expensive test like t0027\nthat takes something like 14 minutes to run, and then a test failure near\nthe end: re-running with verbose output will cost another 14 minutes.\n\nHowever, the benefit of allowing to pass flakey tests outweighs the costs\nhere, if you ask me.\n\n> I'm not overly concerned about the cost of re-running a test, since the\n> idea is that failed tests should be rare. I would be a little worried\n> about flaky tests mysteriously righting themselves on the second run (so\n> you know a failure happened, but you have no good output to describe\n> it).\n\nThis is actually a benefit.\n\nPretty much all the times I can remember a test being flakey (with the one\nnotable exception of the O_APPEND issue with GIT_TRACE), the *tests* were\nbuggy.\n\nPlus, it is less concerning if a test fails occasionally vs a test that\nfails all the time.\n\nMy plan is to indicate the outcome of \"Passed upon rerun\" in the test\noutput eventually, as soon as there is server-side support for it.\n\n(Sidenote: there is technically already server-side support for it, but we\nwould have to generate Visual Studio-style test output, and I had the\nhunch that some die-hard Linuxers among the core Git developers would take\nobjection to that.)\n\n> I do agree that a test_atexit() cleanup would make a lot more sense than\n> what's in the original patch. And that's nicer than the exit trap we're\n> using already, because you may note that each caller has to manually\n> restore the original 'die' handler that test-lib.sh installs.\n\nOkay, then. I will work on this.\n\nCiao,\nDscho\n\n> That would also help with bitrot. If this funky cleanup case only causes\n> problems with junit output, then other people are likely to forget to do\n> it, and the mess falls onto the junit folks (i.e., Dscho). But if the\n> tests start _relying_ on test_atexit() being called (i.e., don't call\n> stop_git_daemon manually, but assume that the atexit handler does so),\n> then the responsible authors are a lot more likely to notice and fix it\n> early.\n> \n> -Peff\n> \n"},{"id":"358150","messageId":"nycvar.QRO.7.76.6.1809141407180.73@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"179314eb-774a-bc98-4a41-f79ec78c8466@gmail.com","subject":"Re: [PATCH 2/9] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-09-14T19:07:34Z","receivedAt":"2018-09-14T19:07:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Sebastian,\n\nOn Wed, 5 Sep 2018, Sebastian Schuberth wrote:\n\n> On 9/3/2018 11:10 PM, Johannes Schindelin via GitGitGadget wrote:\n> \n> > +if test -n \"$TRAVIS_COMMIT\"\n> > +then\n> > +\t# We are running within Travis CI\n> \n> Personally, I'd find a check like\n> \n> if test \"$TRAVIS\" = \"true\"\n> \n> more speaking (also see [1]).\n\nGood call.\n\nWill fix,\nDscho\n\n> \n> [1] https://docs.travis-ci.com/user/environment-variables/\n> \n> -- \n> Sebastian Schuberth\n> \n> \n"},{"id":"360492","messageId":"pull.31.v2.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"[PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:11:57Z","receivedAt":"2018-10-15T10:12:02Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"For a long time already, we have Git's source code continuously tested via\nTravis CI, see e.g. https://travis-ci.org/git/git/builds/421738884. It has\nserved us well, and more and more developers actually pay attention and\nbenefit from the testing this gives us.\n\nIt is also an invaluable tool for contributors who can validate their code\ncontributions via PRs on GitHub, e.g. to verify that their tests do actually\nrun on macOS (i.e. with the BSD family of Unix tools instead of the GNU\none).\n\nThe one sad part about this is the Windows support. Travis lacks it, and we\nwork around that by using Azure Pipelines (the CI part of Azure DevOps,\nformerly known as Visual Studio Team Services) indirectly: one phase in\nTravis would trigger a build, wait for its log, and then paste that log.\n\nAs Git's Windows builds (and tests!) take quite a bit of time, Travis often\ntimed out, or somehow the trigger did not work, and for security reasons\n(the Windows builds are performed in a private pool of containers), the\nWindows builds are completely disabled for Pull Requests on GitHub.\n\nOne might ask why we did not use Azure Pipelines directly. There were a\ncouple of reasons for that:\n\n * most notably, Azure Pipelines' build logs could not be viewed\n anonymously,\n * while Azure Pipelines had Linux and Windows agents, it lacked macOS\n agents,\n * etc\n\nThe main two reasons no longer apply: macOS agents are available now\n[https://docs.microsoft.com/en-us/azure/devops/release-notes/2018/jul-10-vsts]\n, and there is a limited preview of \"public projects\"\n[https://blogs.msdn.microsoft.com/devops/2018/04/27/vsts-public-projects-limited-preview/]\n, i.e. it is possible to configure a Azure Pipelines project so that anybody\ncan view the logs.\n\nI had secured such a public project for Git for Windows already, and I\nrecently also got one for Git. For now, the latter is hooked up with my\npersonal git.git fork on GitHub, but it is my hope that I convince y'all\nthat these Azure Pipelines builds are a good idea, and then hook it up with \nhttps://github.com/git/git.\n\nAs a special treat, this patch series adds the ability to present the\noutcome of Git's test suite as JUnit-style .xml files. This allows the Azure\nPipelines build to present fun diagrams, trends, and makes it a lot easier\nto drill down to test failures than before. See for example \nhttps://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details\n[https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details] \n(you can click on the label of the failed test, and then see the detailed\noutput in the right pane).\n\nThis patch series took way more time than I had originally planned, but I\nthink that in particular the advanced display of the test results was worth\nit. Please let me know what you think about this.\n\nChanges since v1:\n\n * Removed a superfluous eval.\n * Added the commit that fixes the Travis PR builds targeting master that \n just happens to be tagged (see \n https://travis-ci.org/git/git/jobs/424276413 for an incorrectly-skipped\n build).\n * The commit messages and the cover letter now reflect the name change from\n Visual Studio Team Services to Azure DevOps (and in particular, Azure\n Pipelines for the automated builds).\n * Now we're using test_atexit (which we introduced for that purpose)\n instead of hard-coding kill_p4d and stop_git_daemon.\n * The build should now also succeed for Pull Requests (where secret\n variables are not available, for security reasons, and as a consequence\n the file share cannot be mounted).\n * The shell scripted parts now use proper && chains.\n\nJohannes Schindelin (13):\n ci: rename the library of common functions\n ci/lib.sh: encapsulate Travis-specific things\n test-date: add a subcommand to measure times in shell scripts\n tests: optionally write results as JUnit-style .xml\n ci/lib.sh: add support for Azure Pipelines\n Add a build definition for Azure DevOps\n tests: introduce `test_atexit`\n git-daemon: use `test_atexit` in the tests\n git-p4: use `test_atexit` to kill the daemon\n tests: include detailed trace logs with --write-junit-xml upon failure\n tests: record more stderr with --write-junit-xml in case of failure\n README: add a build badge (status of the Azure Pipelines build)\n travis: fix skipping tagged releases\n\n README.md | 2 +\n azure-pipelines.yml | 319 +++++++++++++++++++++\n ci/install-dependencies.sh | 5 +-\n ci/{lib-travisci.sh => lib.sh} | 67 ++++-\n ci/mount-fileshare.sh | 26 ++\n ci/print-test-failures.sh | 4 +-\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 3 +-\n t/.gitignore | 1 +\n t/helper/test-date.c | 12 +\n t/interop/i5500-git-daemon.sh | 1 -\n t/lib-git-daemon.sh | 3 +-\n t/lib-git-p4.sh | 10 +-\n t/t0000-basic.sh | 20 ++\n t/t5570-git-daemon.sh | 1 -\n t/t9800-git-p4-basic.sh | 4 -\n t/t9801-git-p4-branch.sh | 4 -\n t/t9802-git-p4-filetype.sh | 4 -\n t/t9803-git-p4-shell-metachars.sh | 4 -\n t/t9804-git-p4-label.sh | 4 -\n t/t9805-git-p4-skip-submit-edit.sh | 4 -\n t/t9806-git-p4-options.sh | 5 -\n t/t9807-git-p4-submit.sh | 4 -\n t/t9808-git-p4-chdir.sh | 4 -\n t/t9809-git-p4-client-view.sh | 4 -\n t/t9810-git-p4-rcs.sh | 4 -\n t/t9811-git-p4-label-import.sh | 5 -\n t/t9812-git-p4-wildcards.sh | 4 -\n t/t9813-git-p4-preserve-users.sh | 4 -\n t/t9814-git-p4-rename.sh | 4 -\n t/t9815-git-p4-submit-fail.sh | 4 -\n t/t9816-git-p4-locked.sh | 4 -\n t/t9817-git-p4-exclude.sh | 4 -\n t/t9818-git-p4-block.sh | 4 -\n t/t9819-git-p4-case-folding.sh | 4 -\n t/t9820-git-p4-editor-handling.sh | 4 -\n t/t9821-git-p4-path-variations.sh | 4 -\n t/t9822-git-p4-path-encoding.sh | 4 -\n t/t9823-git-p4-mock-lfs.sh | 4 -\n t/t9824-git-p4-git-lfs.sh | 4 -\n t/t9825-git-p4-handle-utf16-without-bom.sh | 4 -\n t/t9826-git-p4-keep-empty-commits.sh | 4 -\n t/t9827-git-p4-change-filetype.sh | 4 -\n t/t9828-git-p4-map-user.sh | 4 -\n t/t9829-git-p4-jobs.sh | 4 -\n t/t9830-git-p4-symlink-dir.sh | 4 -\n t/t9831-git-p4-triggers.sh | 4 -\n t/t9832-unshelve.sh | 4 -\n t/t9833-errors.sh | 5 -\n t/test-lib-functions.sh | 29 ++\n t/test-lib.sh | 140 +++++++++\n 54 files changed, 616 insertions(+), 174 deletions(-)\n create mode 100644 azure-pipelines.yml\n rename ci/{lib-travisci.sh => lib.sh} (61%)\n create mode 100755 ci/mount-fileshare.sh\n\n\nbase-commit: 5a0cc8aca797dbd7d2be3b67458ff880ed45cddf\nPublished-As: https://github.com/gitgitgadget/git/releases/tags/pr-31%2Fdscho%2Fvsts-ci-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-31/dscho/vsts-ci-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/31\n\nRange-diff vs v1:\n\n 1: 858b80bfcf ! 1: c963184510 ci: rename the library of common functions\n @@ -5,8 +5,8 @@\n The name is hard-coded to reflect that we use Travis CI for continuous\n testing.\n \n - In the next commits, we will extend this to be able use Visual Studio\n - Team Services, too.\n + In the next commits, we will extend this to be able use Azure DevOps,\n + too.\n \n So let's adjust the name to make it more generic.\n \n 2: 18e6beec5f ! 2: 815152e0f5 ci/lib.sh: encapsulate Travis-specific things\n @@ -2,8 +2,9 @@\n \n ci/lib.sh: encapsulate Travis-specific things\n \n - The upcoming patches will allow building git.git via VSTS CI, where\n - variable names and URLs look a bit different than in Travis CI.\n + The upcoming patches will allow building git.git via Azure Pipelines\n + (i.e. Azure DevOps' Continuous Integration), where variable names and\n + URLs look a bit different than in Travis CI.\n \n Signed-off-by: Johannes Schindelin \n \n @@ -16,7 +17,7 @@\n \t# brew install gnu-time\n -\tbrew install git-lfs gettext\n +\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n -+\teval brew install $BREW_INSTALL_PACKAGES\n ++\tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \tbrew install caskroom/cask/perforce\n \t;;\n @@ -27,7 +28,7 @@\n @@\n # Library of functions shared by all CI scripts\n \n -+if test -n \"$TRAVIS_COMMIT\"\n ++if test true = \"$TRAVIS\"\n +then\n +\t# We are running within Travis CI\n +\tCI_BRANCH=\"$TRAVIS_BRANCH\"\n 3: 064db77669 = 3: 52337f1875 test-date: add a subcommand to measure times in shell scripts\n 4: 2b5d678594 ! 4: cf4c5ae470 tests: optionally write results as JUnit-style .xml\n @@ -3,7 +3,7 @@\n tests: optionally write results as JUnit-style .xml\n \n This will come in handy when publishing the results of Git's test suite\n - during an automated VSTS CI run.\n + during an automated Azure DevOps run.\n \n Signed-off-by: Johannes Schindelin \n \n 5: 127dfcfb09 ! 5: 486d1d2518 ci/lib.sh: add support for VSTS CI\n @@ -1,6 +1,6 @@\n Author: Johannes Schindelin \n \n - ci/lib.sh: add support for VSTS CI\n + ci/lib.sh: add support for Azure Pipelines\n \n This patch introduces a conditional arm that defines some environment\n variables and a function that displays the URL given the job id (to\n @@ -17,7 +17,7 @@\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n +elif test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n +then\n -+\t# We are running in VSTS CI\n ++\t# We are running in Azure Pipelines\n +\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n +\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n +\tCI_JOB_ID=\"$BUILD_BUILDID\"\n @@ -29,7 +29,7 @@\n +\n +\t# use a subdirectory of the cache dir (because the file share is shared\n +\t# among *all* phases)\n -+\tcache_dir=\"$HOME/vsts-cache/$SYSTEM_PHASENAME\"\n ++\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n +\n +\turl_for_job_id () {\n +\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n 6: ab66060c57 ! 6: 1a22efe849 Add a build definition for VSTS\n @@ -1,8 +1,8 @@\n Author: Johannes Schindelin \n \n - Add a build definition for VSTS\n + Add a build definition for Azure DevOps\n \n - This commit adds a .vsts-ci.yml which is Visual Studio Team Services'\n + This commit adds an azure-pipelines.yml file which is Azure DevOps'\n equivalent to Travis CI's .travis.yml.\n \n To make things a bit easier to understand, we refrain from using the\n @@ -19,10 +19,10 @@\n \n Signed-off-by: Johannes Schindelin \n \n -diff --git a/.vsts-ci.yml b/.vsts-ci.yml\n +diff --git a/azure-pipelines.yml b/azure-pipelines.yml\n new file mode 100644\n --- /dev/null\n -+++ b/.vsts-ci.yml\n ++++ b/azure-pipelines.yml\n @@\n +resources:\n +- repo: self\n @@ -36,12 +36,12 @@\n + name: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n -+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n -+ sudo apt-get update\n -+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin\n ++ sudo apt-get update &&\n ++ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin &&\n +\n -+ export CC=clang\n ++ export CC=clang || exit 1\n +\n + ci/install-dependencies.sh\n + ci/run-build-and-tests.sh || {\n @@ -49,8 +49,10 @@\n + exit 1\n + }\n +\n -+ sudo umount \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n + displayName: 'ci/run-build-and-tests.sh'\n ++ env:\n ++ GITFILESHAREPWD: $(gitfileshare.pwd)\n + - task: PublishTestResults@2\n + displayName: 'Publish Test Results **/TEST-*.xml'\n + inputs:\n @@ -67,10 +69,10 @@\n + name: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n -+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n -+ sudo apt-get update\n -+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin\n ++ sudo apt-get update &&\n ++ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin || exit 1\n +\n + ci/install-dependencies.sh\n + ci/run-build-and-tests.sh || {\n @@ -78,8 +80,10 @@\n + exit 1\n + }\n +\n -+ sudo umount \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n + displayName: 'ci/run-build-and-tests.sh'\n ++ env:\n ++ GITFILESHAREPWD: $(gitfileshare.pwd)\n + - task: PublishTestResults@2\n + displayName: 'Publish Test Results **/TEST-*.xml'\n + inputs:\n @@ -96,7 +100,7 @@\n + name: Hosted macOS\n + steps:\n + - bash: |\n -+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n + export CC=clang\n +\n @@ -106,8 +110,10 @@\n + exit 1\n + }\n +\n -+ umount \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n + displayName: 'ci/run-build-and-tests.sh'\n ++ env:\n ++ GITFILESHAREPWD: $(gitfileshare.pwd)\n + - task: PublishTestResults@2\n + displayName: 'Publish Test Results **/TEST-*.xml'\n + inputs:\n @@ -124,7 +130,7 @@\n + name: Hosted macOS\n + steps:\n + - bash: |\n -+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n + ci/install-dependencies.sh\n + ci/run-build-and-tests.sh || {\n @@ -132,8 +138,10 @@\n + exit 1\n + }\n +\n -+ umount \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n + displayName: 'ci/run-build-and-tests.sh'\n ++ env:\n ++ GITFILESHAREPWD: $(gitfileshare.pwd)\n + - task: PublishTestResults@2\n + displayName: 'Publish Test Results **/TEST-*.xml'\n + inputs:\n @@ -150,20 +158,22 @@\n + name: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n -+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n -+ sudo apt-get update\n -+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev\n ++ sudo apt-get update &&\n ++ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev &&\n +\n -+ export jobname=GETTEXT_POISON\n ++ export jobname=GETTEXT_POISON || exit 1\n +\n + ci/run-build-and-tests.sh || {\n + ci/print-test-failures.sh\n + exit 1\n + }\n +\n -+ sudo umount \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n + displayName: 'ci/run-build-and-tests.sh'\n ++ env:\n ++ GITFILESHAREPWD: $(gitfileshare.pwd)\n + - task: PublishTestResults@2\n + displayName: 'Publish Test Results **/TEST-*.xml'\n + inputs:\n @@ -184,8 +194,10 @@\n + # Helper to check the error level of the latest command (exit with error when appropriate)\n + function c() { if (!$?) { exit(1) } }\n +\n -+ net use s: \\\\gitfileshare.file.core.windows.net\\vsts-cache \"$(gitfileshare.pwd)\" /user:AZURE\\gitfileshare /persistent:no; c\n -+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\vsts-cache\" S:\\; c\n ++ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n ++ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no; c\n ++ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\; c\n ++ }\n +\n + # Add build agent's MinGit to PATH\n + $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n @@ -215,7 +227,7 @@\n +\n + # Initialize Git for Windows' SDK\n + $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n -+ init \"$sdk_path\" \"https://git-for-windows.visualstudio.com/_git/git-sdk-64\" 0\n ++ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n + init usr\\src\\build-extra https://github.com/git-for-windows/build-extra 1\n +\n + cd \"$(Build.SourcesDirectory)\"; c\n @@ -227,14 +239,19 @@\n +\n + make -j10 DEVELOPER=1 NO_PERL=1 || exit 1\n + NO_PERL=1 NO_SVN_TESTS=1 GIT_TEST_OPTS=\\\"--quiet --write-junit-xml\\\" time make -j15 -k DEVELOPER=1 test || {\n -+ NO_PERL=1 NO_SVN_TESTS=1 GIT_TEST_OPTS=\\\"-i -v -x\\\" make -k failed; exit 1\n ++ NO_PERL=1 NO_SVN_TESTS=1 GIT_TEST_OPTS=\\\"-i -v -x\\\" make -k -C t failed; exit 1\n + }\n +\n + save_good_tree\n + \"@\n ++ c\n +\n -+ cmd /c rmdir \"$(Build.SourcesDirectory)\\vsts-cache\"\n ++ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n ++ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n ++ }\n + displayName: 'build & test'\n ++ env:\n ++ GITFILESHAREPWD: $(gitfileshare.pwd)\n + - task: PublishTestResults@2\n + displayName: 'Publish Test Results **/TEST-*.xml'\n + inputs:\n @@ -251,28 +268,30 @@\n + name: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n -+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n -+ sudo apt-get update\n ++ sudo apt-get update &&\n + sudo apt-get -y install \\\n + apt-transport-https \\\n + ca-certificates \\\n + curl \\\n -+ software-properties-common\n -+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -\n ++ software-properties-common &&\n ++ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - &&\n + sudo add-apt-repository \\\n + \"deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n + $(lsb_release -cs) \\\n -+ stable\"\n -+ sudo apt-get update\n -+ sudo apt-get -y install docker-ce\n ++ stable\" &&\n ++ sudo apt-get update &&\n ++ sudo apt-get -y install docker-ce &&\n +\n + sudo AGENT_OS=\"$AGENT_OS\" BUILD_BUILDNUMBER=\"$BUILD_BUILDNUMBER\" BUILD_REPOSITORY_URI=\"$BUILD_REPOSITORY_URI\" BUILD_SOURCEBRANCH=\"$BUILD_SOURCEBRANCH\" BUILD_SOURCEVERSION=\"$BUILD_SOURCEVERSION\" SYSTEM_PHASENAME=\"$SYSTEM_PHASENAME\" SYSTEM_TASKDEFINITIONSURI=\"$SYSTEM_TASKDEFINITIONSURI\" SYSTEM_TEAMPROJECT=\"$SYSTEM_TEAMPROJECT\" CC=$CC MAKEFLAGS=-j3 bash -lxc ci/run-linux32-docker.sh || exit 1\n +\n + sudo chmod a+r t/out/TEST-*.xml\n +\n -+ sudo umount \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n + displayName: 'ci/run-linux32-docker.sh'\n ++ env:\n ++ GITFILESHAREPWD: $(gitfileshare.pwd)\n + - task: PublishTestResults@2\n + displayName: 'Publish Test Results **/TEST-*.xml'\n + inputs:\n @@ -289,17 +308,19 @@\n + name: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n -+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n -+ sudo apt-get update\n -+ sudo apt-get install -y coccinelle\n ++ sudo apt-get update &&\n ++ sudo apt-get install -y coccinelle &&\n +\n -+ export jobname=StaticAnalysis\n ++ export jobname=StaticAnalysis &&\n +\n + ci/run-static-analysis.sh || exit 1\n +\n -+ sudo umount \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n + displayName: 'ci/run-static-analysis.sh'\n ++ env:\n ++ GITFILESHAREPWD: $(gitfileshare.pwd)\n +\n +- phase: documentation\n + displayName: Documentation\n @@ -308,18 +329,20 @@\n + name: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n -+ ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/vsts-cache gitfileshare \"$(gitfileshare.pwd)\" \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n -+ sudo apt-get update\n -+ sudo apt-get install -y asciidoc xmlto asciidoctor\n ++ sudo apt-get update &&\n ++ sudo apt-get install -y asciidoc xmlto asciidoctor &&\n +\n -+ export ALREADY_HAVE_ASCIIDOCTOR=yes.\n -+ export jobname=Documentation\n ++ export ALREADY_HAVE_ASCIIDOCTOR=yes. &&\n ++ export jobname=Documentation &&\n +\n + ci/test-documentation.sh || exit 1\n +\n -+ sudo umount \"$HOME/vsts-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n + displayName: 'ci/test-documentation.sh'\n ++ env:\n ++ GITFILESHAREPWD: $(gitfileshare.pwd)\n \n diff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\n new file mode 100755\n -: ---------- > 7: 12d6137f8d tests: introduce `test_atexit`\n -: ---------- > 8: 3bb226b79b git-daemon: use `test_atexit` in the tests\n -: ---------- > 9: 3e2193a73d git-p4: use `test_atexit` to kill the daemon\n 7: 942bf423a4 ! 10: ae3c42519a tests: include detailed trace logs with --write-junit-xml upon failure\n @@ -14,9 +14,9 @@\n almost all of those cases: only when a test fails do we have a way to\n include the trace.\n \n - So let's do something different in VSTS: let's run all the tests with\n - `--quiet` first, and only if a failure is encountered, try to trace the\n - commands as they are executed.\n + So let's do something different when using Azure DevOps: let's run all\n + the tests with `--quiet` first, and only if a failure is encountered,\n + try to trace the commands as they are executed.\n \n Of course, we cannot turn on `--verbose-log` after the fact. So let's\n just re-run the test with all the same options, adding `--verbose-log`.\n @@ -25,15 +25,10 @@\n Note: there is an off chance that re-running the test in verbose mode\n \"fixes\" the failures (and this does happen from time to time!). That is\n a possibility we should be able to live with. Ideally, we would label\n - this as \"Passed upon rerun\", and that outcome even exists on VSTS, but\n - it is not available when using the JUnit XML format for now:\n - https://github.com/Microsoft/vsts-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n -\n - This patch contains a slightly inelegant workaround for the p4 and\n - git-daemon tests: when we try to re-run the p4/git-daemon tests after\n - the daemon has been started already, we need to kill said daemon. We do\n - this by detecting the presence of the `kill_p4d` and `stop_git_daemon`\n - shell functions and calling them if available.\n + this as \"Passed upon rerun\", and Azure Pipelines even know about that\n + outcome, but it is not available when using the JUnit XML format for\n + now:\n + https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n \n Signed-off-by: Johannes Schindelin \n \n @@ -60,14 +55,8 @@\n \tthen\n +\t\tif test -z \"$GIT_TEST_TEE_OUTPUT_FILE\"\n +\t\tthen\n -+\t\t\tcase \"$(type kill_p4d 2>/dev/null | head -n 1)\" in\n -+\t\t\t*function*) kill_p4d;;\n -+\t\t\tesac\n -+\n -+\t\t\tcase \"$(type stop_git_daemon 2>/dev/null |\n -+\t\t\t\thead -n 1)\" in\n -+\t\t\t*function*) stop_git_daemon;;\n -+\t\t\tesac\n ++\t\t\t# clean up\n ++\t\t\ttest_atexit_handler\n +\n +\t\t\t# re-run with --verbose-log\n +\t\t\techo \"# Re-running: $junit_rerun_options_sq\" >&2\n 8: 3e83c64090 = 11: 2466a48aa3 tests: record more stderr with --write-junit-xml in case of failure\n 9: dc1d890d71 ! 12: d112b3fe86 README: add a build badge (status of the VSTS build)\n @@ -1,6 +1,6 @@\n Author: Johannes Schindelin \n \n - README: add a build badge (status of the VSTS build)\n + README: add a build badge (status of the Azure Pipelines build)\n \n Just like so many other OSS projects, we now also have a build badge.\n \n @@ -10,7 +10,7 @@\n --- a/README.md\n +++ b/README.md\n @@\n -+[![Build Status](https://git.visualstudio.com/git/_apis/build/status/test-git.git)](https://git.visualstudio.com/git/_build/latest?definitionId=2)\n ++[![Build Status](https:/dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2)\n +\n Git - fast, scalable, distributed revision control system\n =========================================================\n -: ---------- > 13: 0a53f37135 travis: fix skipping tagged releases\n\n-- \ngitgitgadget\n"},{"id":"360493","messageId":"c9631845104dd66e1be6df9f15eec2481089c85f.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 01/13] ci: rename the library of common functions","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:11:59Z","receivedAt":"2018-10-15T10:12:04Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe name is hard-coded to reflect that we use Travis CI for continuous\ntesting.\n\nIn the next commits, we will extend this to be able use Azure DevOps,\ntoo.\n\nSo let's adjust the name to make it more generic.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 2 +-\n ci/{lib-travisci.sh => lib.sh} | 0\n ci/print-test-failures.sh | 2 +-\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 2 +-\n 8 files changed, 7 insertions(+), 7 deletions(-)\n rename ci/{lib-travisci.sh => lib.sh} (100%)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 75a9fd2475..961064658e 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -3,7 +3,7 @@\n # Install dependencies required to build and test Git on Linux and macOS\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\ndiff --git a/ci/lib-travisci.sh b/ci/lib.sh\nsimilarity index 100%\nrename from ci/lib-travisci.sh\nrename to ci/lib.sh\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex d55460a212..7aef39a2fd 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -3,7 +3,7 @@\n # Print output of failing tests\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n # Tracing executed commands would produce too much noise in the loop below.\n set +x\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 2a5bff4a1c..e28ac2fb9a 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -3,7 +3,7 @@\n # Build and test Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n ln -s \"$cache_dir/.prove\" t/.prove\n \ndiff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh\nindex 21637903ce..751acfcf8a 100755\n--- a/ci/run-linux32-docker.sh\n+++ b/ci/run-linux32-docker.sh\n@@ -3,7 +3,7 @@\n # Download and run Docker image to build and test 32-bit Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n docker pull daald/ubuntu32:xenial\n \ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\nindex 5688f261d0..dc189c7456 100755\n--- a/ci/run-static-analysis.sh\n+++ b/ci/run-static-analysis.sh\n@@ -3,7 +3,7 @@\n # Perform various static code analysis checks\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n make --jobs=2 coccicheck\n \ndiff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh\nindex d99a180e52..a73a4eca0a 100755\n--- a/ci/run-windows-build.sh\n+++ b/ci/run-windows-build.sh\n@@ -6,7 +6,7 @@\n # supported) and a commit hash.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n test $# -ne 2 && echo \"Unexpected number of parameters\" && exit 1\n test -z \"$GFW_CI_TOKEN\" && echo \"GFW_CI_TOKEN not defined\" && exit\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex a20de9ca12..d3cdbac73f 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -3,7 +3,7 @@\n # Perform sanity checks on documentation and build it.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n gem install asciidoctor\n \n-- \ngitgitgadget\n\n"},{"id":"360494","messageId":"815152e0f57d545ae2fae7429c16e5be497746be.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 02/13] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:00Z","receivedAt":"2018-10-15T10:12:04Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe upcoming patches will allow building git.git via Azure Pipelines\n(i.e. Azure DevOps' Continuous Integration), where variable names and\nURLs look a bit different than in Travis CI.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 3 ++-\n ci/lib.sh | 44 +++++++++++++++++++++++++++-----------\n ci/print-test-failures.sh | 2 +-\n ci/test-documentation.sh | 1 +\n 4 files changed, 35 insertions(+), 15 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 961064658e..63fa37f68f 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -28,7 +28,8 @@ osx-clang|osx-gcc)\n \tbrew update --quiet\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\n-\tbrew install git-lfs gettext\n+\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n+\tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \tbrew install caskroom/cask/perforce\n \t;;\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 06970f7213..8532555b4e 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,5 +1,26 @@\n # Library of functions shared by all CI scripts\n \n+if test true = \"$TRAVIS\"\n+then\n+\t# We are running within Travis CI\n+\tCI_BRANCH=\"$TRAVIS_BRANCH\"\n+\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n+\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n+\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n+\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n+\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n+\n+\tcache_dir=\"$HOME/travis-cache\"\n+\n+\turl_for_job_id () {\n+\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+fi\n+\n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n \t# at the same commit as the tip of the branch is pushed, and building\n@@ -13,10 +34,10 @@ skip_branch_tip_with_tag () {\n \t# we can skip the build because we won't be skipping a build\n \t# of a tag.\n \n-\tif TAG=$(git describe --exact-match \"$TRAVIS_BRANCH\" 2>/dev/null) &&\n-\t\ttest \"$TAG\" != \"$TRAVIS_BRANCH\"\n+\tif TAG=$(git describe --exact-match \"$CI_BRANCH\" 2>/dev/null) &&\n+\t\ttest \"$TAG\" != \"$CI_BRANCH\"\n \tthen\n-\t\techo \"$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)\"\n+\t\techo \"$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)\"\n \t\texit 0\n \tfi\n }\n@@ -25,7 +46,7 @@ skip_branch_tip_with_tag () {\n # job if we encounter the same tree again and can provide a useful info\n # message.\n save_good_tree () {\n-\techo \"$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID\" >>\"$good_trees_file\"\n+\techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n \t# limit the file size\n \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n \tmv \"$good_trees_file\".tmp \"$good_trees_file\"\n@@ -35,7 +56,7 @@ save_good_tree () {\n # successfully before (e.g. because the branch got rebased, changing only\n # the commit messages).\n skip_good_tree () {\n-\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $TRAVIS_COMMIT^{tree}) \" \"$good_trees_file\")\"\n+\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n \tthen\n \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n \t\t# Continue the build job.\n@@ -45,18 +66,18 @@ skip_good_tree () {\n \techo \"$good_tree_info\" | {\n \t\tread tree prev_good_commit prev_good_job_number prev_good_job_id\n \n-\t\tif test \"$TRAVIS_JOB_ID\" = \"$prev_good_job_id\"\n+\t\tif test \"$CI_JOB_ID\" = \"$prev_good_job_id\"\n \t\tthen\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit has already been built and tested successfully by this build job.\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\telse\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.\n-\t\t\tThe log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id\n+\t\t\tThe log of that build job is available at $(url_for_job_id $prev_good_job_id)\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\tfi\n@@ -81,7 +102,6 @@ check_unignored_build_artifacts ()\n # and installing dependencies.\n set -ex\n \n-cache_dir=\"$HOME/travis-cache\"\n good_trees_file=\"$cache_dir/good-trees\"\n \n mkdir -p \"$cache_dir\"\n@@ -91,13 +111,11 @@ skip_good_tree\n \n if test -z \"$jobname\"\n then\n-\tjobname=\"$TRAVIS_OS_NAME-$CC\"\n+\tjobname=\"$CI_OS_NAME-$CC\"\n fi\n \n export DEVELOPER=1\n export DEFAULT_TEST_TARGET=prove\n-export GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n-export GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n export GIT_TEST_CLONE_2GB=YesPlease\n if [ \"$jobname\" = linux-gcc ]; then\n \texport CC=gcc-8\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 7aef39a2fd..d2045b63a6 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -69,7 +69,7 @@ do\n \tfi\n done\n \n-if [ $combined_trash_size -gt 0 ]\n+if [ -n \"$TRAVIS_JOB_ID\" -a $combined_trash_size -gt 0 ]\n then\n \techo \"------------------------------------------------------------------------\"\n \techo \"Trash directories embedded in this log can be extracted by running:\"\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex d3cdbac73f..7d0beb2832 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -5,6 +5,7 @@\n \n . ${0%/*}/lib.sh\n \n+test -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n gem install asciidoctor\n \n make check-builtins\n-- \ngitgitgadget\n\n"},{"id":"360495","messageId":"52337f18750e16a9ba3bb11e9c2f3748f438846c.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 03/13] test-date: add a subcommand to measure times in shell scripts","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:02Z","receivedAt":"2018-10-15T10:12:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIn the next commit, we want to teach Git's test suite to optionally\noutput test results in JUnit-style .xml files. These files contain\ninformation about the time spent. So we need a way to measure time.\n\nWhile we could use `date +%s` for that, this will give us only seconds,\ni.e. very coarse-grained timings.\n\nGNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output),\nbut there is no equivalent in BSD `date` (read: on macOS, we would not\nbe able to obtain precise timings).\n\nSo let's introduce `test-tool date getnanos`, with an optional start\ntime, that outputs preciser values.\n\nGranted, it is a bit pointless to try measuring times accurately in\nshell scripts, certainly to nanosecond precision. But it is better than\nsecond-granularity.\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-date.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex a0837371ab..792a805374 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -7,6 +7,7 @@ static const char *usage_msg = \"\\n\"\n \" test-tool date parse [date]...\\n\"\n \" test-tool date approxidate [date]...\\n\"\n \" test-tool date timestamp [date]...\\n\"\n+\" test-tool date getnanos [start-nanos]\\n\"\n \" test-tool date is64bit\\n\"\n \" test-tool date time_t-is64bit\\n\";\n \n@@ -82,6 +83,15 @@ static void parse_approx_timestamp(const char **argv, struct timeval *now)\n \t}\n }\n \n+static void getnanos(const char **argv, struct timeval *now)\n+{\n+\tdouble seconds = getnanotime() / 1.0e9;\n+\n+\tif (*argv)\n+\t\tseconds -= strtod(*argv, NULL);\n+\tprintf(\"%lf\\n\", seconds);\n+}\n+\n int cmd__date(int argc, const char **argv)\n {\n \tstruct timeval now;\n@@ -108,6 +118,8 @@ int cmd__date(int argc, const char **argv)\n \t\tparse_approxidate(argv+1, &now);\n \telse if (!strcmp(*argv, \"timestamp\"))\n \t\tparse_approx_timestamp(argv+1, &now);\n+\telse if (!strcmp(*argv, \"getnanos\"))\n+\t\tgetnanos(argv+1, &now);\n \telse if (!strcmp(*argv, \"is64bit\"))\n \t\treturn sizeof(timestamp_t) == 8 ? 0 : 1;\n \telse if (!strcmp(*argv, \"time_t-is64bit\"))\n-- \ngitgitgadget\n\n"},{"id":"360496","messageId":"cf4c5ae47007651a95cbfc56f126326429e17999.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 04/13] tests: optionally write results as JUnit-style .xml","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:04Z","receivedAt":"2018-10-15T10:12:07Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis will come in handy when publishing the results of Git's test suite\nduring an automated Azure DevOps run.\n\nSigned-off-by: Johannes Schindelin \n---\n t/.gitignore | 1 +\n t/test-lib.sh | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 99 insertions(+)\n\ndiff --git a/t/.gitignore b/t/.gitignore\nindex 348715f0e4..91cf5772fe 100644\n--- a/t/.gitignore\n+++ b/t/.gitignore\n@@ -2,3 +2,4 @@\n /test-results\n /.prove\n /chainlinttmp\n+/out/\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 3f95bfda60..7ed0013f6d 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -288,6 +288,9 @@ do\n \t--verbose-log)\n \t\tverbose_log=t\n \t\tshift ;;\n+\t--write-junit-xml)\n+\t\twrite_junit_xml=t\n+\t\tshift ;;\n \t*)\n \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n \tesac\n@@ -431,11 +434,24 @@ trap 'exit $?' INT\n # the test_expect_* functions instead.\n \n test_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$*\"\n+\tfi\n \ttest_success=$(($test_success + 1))\n \tsay_color \"\" \"ok $test_count - $@\"\n }\n \n test_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_insert=\"\"\n+\t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n+\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\tjunit_insert=\"$junit_insert\"\n+\t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n+\tfi\n \ttest_failure=$(($test_failure + 1))\n \tsay_color error \"not ok $test_count - $1\"\n \tshift\n@@ -444,11 +460,19 @@ test_failure_ () {\n }\n \n test_known_broken_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (breakage fixed)\"\n+\tfi\n \ttest_fixed=$(($test_fixed+1))\n \tsay_color error \"ok $test_count - $@ # TODO known breakage vanished\"\n }\n \n test_known_broken_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (known breakage)\"\n+\tfi\n \ttest_broken=$(($test_broken+1))\n \tsay_color warn \"not ok $test_count - $@ # TODO known breakage\"\n }\n@@ -706,6 +730,10 @@ test_start_ () {\n \ttest_count=$(($test_count+1))\n \tmaybe_setup_verbose\n \tmaybe_setup_valgrind\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_start=$(test-tool date getnanos)\n+\tfi\n }\n \n test_finish_ () {\n@@ -743,6 +771,13 @@ test_skip () {\n \n \tcase \"$to_skip\" in\n \tt)\n+\t\tif test -n \"$write_junit_xml\"\n+\t\tthen\n+\t\t\tmessage=\"$(xml_attr_encode \"$skipped_reason\")\"\n+\t\t\twrite_junit_xml_testcase \"$1\" \\\n+\t\t\t\t\" \"\n+\t\tfi\n+\n \t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n@@ -758,9 +793,58 @@ test_at_end_hook_ () {\n \t:\n }\n \n+write_junit_xml () {\n+\tcase \"$1\" in\n+\t--truncate)\n+\t\t>\"$junit_xml_path\"\n+\t\tjunit_have_testcase=\n+\t\tshift\n+\t\t;;\n+\tesac\n+\tprintf '%s\\n' \"$@\" >>\"$junit_xml_path\"\n+}\n+\n+xml_attr_encode () {\n+\t# We do not translate CR to because BSD sed does not handle\n+\t# \\r in the regex. In practice, the output should not even have any\n+\t# carriage returns.\n+\tprintf '%s\\n' \"$@\" |\n+\tsed -e 's/&/\\&/g' -e \"s/'/\\'/g\" -e 's/\"/\\"/g' \\\n+\t\t-e 's//\\>/g' \\\n+\t\t-e 's/\t/\\ /g' -e 's/$/\\ /' -e '$s/ $//' |\n+\ttr -d '\\012\\015'\n+}\n+\n+write_junit_xml_testcase () {\n+\tjunit_attrs=\"name=\\\"$(xml_attr_encode \"$this_test.$test_count $1\")\\\"\"\n+\tshift\n+\tjunit_attrs=\"$junit_attrs classname=\\\"$this_test\\\"\"\n+\tjunit_attrs=\"$junit_attrs time=\\\"$(test-tool \\\n+\t\tdate getnanos $junit_start)\\\"\"\n+\twrite_junit_xml \"$(printf '%s\\n' \\\n+\t\t\" \" \"$@\" \" \")\"\n+\tjunit_have_testcase=t\n+}\n+\n test_done () {\n \tGIT_EXIT_OK=t\n \n+\tif test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n+\tthen\n+\t\ttest -n \"$junit_have_testcase\" || {\n+\t\t\tjunit_start=$(test-tool date getnanos)\n+\t\t\twrite_junit_xml_testcase \"all tests skipped\"\n+\t\t}\n+\n+\t\t# adjust the overall time\n+\t\tjunit_time=$(test-tool date getnanos $junit_suite_start)\n+\t\tsed \"s/]*/& time=\\\"$junit_time\\\"/\" \\\n+\t\t\t<\"$junit_xml_path\" >\"$junit_xml_path.new\"\n+\t\tmv \"$junit_xml_path.new\" \"$junit_xml_path\"\n+\n+\t\twrite_junit_xml \" \" \"\"\n+\tfi\n+\n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n \t\ttest_results_dir=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n@@ -996,6 +1080,7 @@ then\n else\n \tmkdir -p \"$TRASH_DIRECTORY\"\n fi\n+\n # Use -P to resolve symlinks in our working directory so that the cwd\n # in subprocesses like git equals our $PWD (for pathname comparisons).\n cd -P \"$TRASH_DIRECTORY\" || exit 1\n@@ -1009,6 +1094,19 @@ then\n \ttest_done\n fi\n \n+if test -n \"$write_junit_xml\"\n+then\n+\tjunit_xml_dir=\"$TEST_OUTPUT_DIRECTORY/out\"\n+\tmkdir -p \"$junit_xml_dir\"\n+\tjunit_xml_base=${0##*/}\n+\tjunit_xml_path=\"$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml\"\n+\tjunit_attrs=\"name=\\\"${junit_xml_base%.sh}\\\"\"\n+\tjunit_attrs=\"$junit_attrs timestamp=\\\"$(TZ=UTC \\\n+\t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n+\twrite_junit_xml --truncate \"\" \" \"\n+\tjunit_suite_start=$(test-tool date getnanos)\n+fi\n+\n # Provide an implementation of the 'yes' utility\n yes () {\n \tif test $# = 0\n-- \ngitgitgadget\n\n"},{"id":"360497","messageId":"486d1d25183fc7b0de599a0791ed41a87cc176cb.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 05/13] ci/lib.sh: add support for Azure Pipelines","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:05Z","receivedAt":"2018-10-15T10:12:08Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis patch introduces a conditional arm that defines some environment\nvariables and a function that displays the URL given the job id (to\nidentify previous runs for known-good trees).\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 8532555b4e..584abcd529 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -19,6 +19,29 @@ then\n \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+elif test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n+then\n+\t# We are running in Azure Pipelines\n+\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n+\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n+\tCI_JOB_ID=\"$BUILD_BUILDID\"\n+\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n+\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n+\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n+\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n+\tCC=\"${CC:-gcc}\"\n+\n+\t# use a subdirectory of the cache dir (because the file share is shared\n+\t# among *all* phases)\n+\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n+\n+\turl_for_job_id () {\n+\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n fi\n \n skip_branch_tip_with_tag () {\n-- \ngitgitgadget\n\n"},{"id":"360498","messageId":"1a22efe849d6da79f2c639c62a1483361a130238.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 06/13] Add a build definition for Azure DevOps","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:06Z","receivedAt":"2018-10-15T10:12:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis commit adds an azure-pipelines.yml file which is Azure DevOps'\nequivalent to Travis CI's .travis.yml.\n\nTo make things a bit easier to understand, we refrain from using the\n`matrix` feature here because (while it is powerful) it can be a bit\nconfusing to users who are not familiar with CI setups. Therefore, we\nuse a separate phase even for similar configurations (such as GCC vs\nClang on Linux, GCC vs Clang on macOS).\n\nAlso, we make use of the shiny new feature we just introduced where the\ntest suite can output JUnit-style .xml files. This information is made\navailable in a nice UI that allows the viewer to filter by phase and/or\ntest number, and to see trends such as: number of (failing) tests, time\nspent running the test suite, etc.\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 319 ++++++++++++++++++++++++++++++++++++++++++\n ci/mount-fileshare.sh | 26 ++++\n 2 files changed, 345 insertions(+)\n create mode 100644 azure-pipelines.yml\n create mode 100755 ci/mount-fileshare.sh\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nnew file mode 100644\nindex 0000000000..b5749121d2\n--- /dev/null\n+++ b/azure-pipelines.yml\n@@ -0,0 +1,319 @@\n+resources:\n+- repo: self\n+ fetchDepth: 1\n+\n+phases:\n+- phase: linux_clang\n+ displayName: linux-clang\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin &&\n+\n+ export CC=clang || exit 1\n+\n+ ci/install-dependencies.sh\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-clang'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: linux_gcc\n+ displayName: linux-gcc\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin || exit 1\n+\n+ ci/install-dependencies.sh\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-gcc'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: osx_clang\n+ displayName: osx-clang\n+ condition: succeeded()\n+ queue:\n+ name: Hosted macOS\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ export CC=clang\n+\n+ ci/install-dependencies.sh\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-clang'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: osx_gcc\n+ displayName: osx-gcc\n+ condition: succeeded()\n+ queue:\n+ name: Hosted macOS\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ ci/install-dependencies.sh\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-gcc'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: gettext_poison\n+ displayName: GETTEXT_POISON\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev &&\n+\n+ export jobname=GETTEXT_POISON || exit 1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'gettext-poison'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: windows\n+ displayName: Windows\n+ condition: succeeded()\n+ queue:\n+ name: Hosted VS2017\n+ timeoutInMinutes: 240\n+ steps:\n+ - powershell: |\n+ # Helper to check the error level of the latest command (exit with error when appropriate)\n+ function c() { if (!$?) { exit(1) } }\n+\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no; c\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\; c\n+ }\n+\n+ # Add build agent's MinGit to PATH\n+ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n+\n+ # Helper to initialize (or update) a Git worktree\n+ function init ($path, $url, $set_origin) {\n+ if (Test-Path $path) {\n+ cd $path; c\n+ if (Test-Path .git) {\n+ git init; c\n+ } else {\n+ git status\n+ }\n+ } else {\n+ git init $path; c\n+ cd $path; c\n+ }\n+ git config core.autocrlf false; c\n+ git config core.untrackedCache true; c\n+ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n+ git remote add origin $url; c\n+ }\n+ git fetch --depth=1 $url master; c\n+ git reset --hard FETCH_HEAD; c\n+ git clean -df; c\n+ }\n+\n+ # Initialize Git for Windows' SDK\n+ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n+ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+ init usr\\src\\build-extra https://github.com/git-for-windows/build-extra 1\n+\n+ cd \"$(Build.SourcesDirectory)\"; c\n+\n+ $env:HOME = \"$(Build.SourcesDirectory)\"\n+ $env:MSYSTEM = \"MINGW64\"\n+ git-sdk-64\\git-cmd --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ . ci/lib.sh\n+\n+ make -j10 DEVELOPER=1 NO_PERL=1 || exit 1\n+ NO_PERL=1 NO_SVN_TESTS=1 GIT_TEST_OPTS=\\\"--quiet --write-junit-xml\\\" time make -j15 -k DEVELOPER=1 test || {\n+ NO_PERL=1 NO_SVN_TESTS=1 GIT_TEST_OPTS=\\\"-i -v -x\\\" make -k -C t failed; exit 1\n+ }\n+\n+ save_good_tree\n+ \"@\n+ c\n+\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'build & test'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'windows'\n+ platform: Windows\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: linux32\n+ displayName: Linux32\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install \\\n+ apt-transport-https \\\n+ ca-certificates \\\n+ curl \\\n+ software-properties-common &&\n+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - &&\n+ sudo add-apt-repository \\\n+ \"deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n+ $(lsb_release -cs) \\\n+ stable\" &&\n+ sudo apt-get update &&\n+ sudo apt-get -y install docker-ce &&\n+\n+ sudo AGENT_OS=\"$AGENT_OS\" BUILD_BUILDNUMBER=\"$BUILD_BUILDNUMBER\" BUILD_REPOSITORY_URI=\"$BUILD_REPOSITORY_URI\" BUILD_SOURCEBRANCH=\"$BUILD_SOURCEBRANCH\" BUILD_SOURCEVERSION=\"$BUILD_SOURCEVERSION\" SYSTEM_PHASENAME=\"$SYSTEM_PHASENAME\" SYSTEM_TASKDEFINITIONSURI=\"$SYSTEM_TASKDEFINITIONSURI\" SYSTEM_TEAMPROJECT=\"$SYSTEM_TEAMPROJECT\" CC=$CC MAKEFLAGS=-j3 bash -lxc ci/run-linux32-docker.sh || exit 1\n+\n+ sudo chmod a+r t/out/TEST-*.xml\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-linux32-docker.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux32'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- phase: static_analysis\n+ displayName: StaticAnalysis\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get install -y coccinelle &&\n+\n+ export jobname=StaticAnalysis &&\n+\n+ ci/run-static-analysis.sh || exit 1\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-static-analysis.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+\n+- phase: documentation\n+ displayName: Documentation\n+ condition: succeeded()\n+ queue:\n+ name: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get install -y asciidoc xmlto asciidoctor &&\n+\n+ export ALREADY_HAVE_ASCIIDOCTOR=yes. &&\n+ export jobname=Documentation &&\n+\n+ ci/test-documentation.sh || exit 1\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/test-documentation.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\ndiff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\nnew file mode 100755\nindex 0000000000..5fb5f74b70\n--- /dev/null\n+++ b/ci/mount-fileshare.sh\n@@ -0,0 +1,26 @@\n+#!/bin/sh\n+\n+die () {\n+\techo \"$*\" >&2\n+\texit 1\n+}\n+\n+test $# = 4 ||\n+die \"Usage: $0 \n\nWhen running the p4 daemon or `git daemon`, we want to kill it at the\nend of the test script.\n\nSo far, we do this \"manually\".\n\nHowever, in the next few commits we want to teach the test suite to\noptionally re-run scripts with different options, therefore we will have\nto have a consistent way to stop daemons.\n\nLet's introduce `test_atexit`, which is loosely modeled after\n`test_when_finished` (but has a broader scope: rather than running the\ncommands after the current test case, run them when the test script\nfinishes, and also run them when the `--immediate` option is in effect).\n\nSigned-off-by: Johannes Schindelin \n---\n t/t0000-basic.sh | 18 ++++++++++++++++++\n t/test-lib-functions.sh | 29 +++++++++++++++++++++++++++++\n t/test-lib.sh | 4 ++++\n 3 files changed, 51 insertions(+)\n\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 391f910c6a..8c5faa6ce1 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -821,6 +821,24 @@ test_expect_success 'tests clean up even on failures' \"\n \tEOF\n \"\n \n+test_expect_success 'test_atexit is run' \"\n+\ttest_must_fail run_sub_test_lib_test \\\n+\t\tatexit-cleanup 'Run atexit commands' -i <<-\\\\EOF &&\n+\ttest_expect_success 'tests clean up even after a failure' '\n+\t\t> ../../clean-atexit &&\n+\t\ttest_atexit rm ../../clean-atexit &&\n+\t\t> ../../also-clean-atexit &&\n+\t\ttest_atexit rm ../../also-clean-atexit &&\n+\t\t> ../../dont-clean-atexit &&\n+\t\t(exit 1)\n+\t'\n+\ttest_done\n+\tEOF\n+\ttest_path_exists dont-clean-atexit &&\n+\ttest_path_is_missing clean-atexit &&\n+\ttest_path_is_missing also-clean-atexit\n+\"\n+\n test_expect_success 'test_oid setup' '\n \ttest_oid_init\n '\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 78d8c3783b..d7dd0c1be9 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -891,6 +891,35 @@ test_when_finished () {\n \t\t} && (exit \\\"\\$eval_ret\\\"); eval_ret=\\$?; $test_cleanup\"\n }\n \n+# This function can be used to schedule some commands to be run\n+# unconditionally at the end of the test script, e.g. to stop a daemon:\n+#\n+#\ttest_expect_success 'test git daemon' '\n+#\t\tgit daemon &\n+#\t\tdaemon_pid=$! &&\n+#\t\ttest_atexit \"kill $daemon_pid\" &&\n+#\t\thello world\n+#\t'\n+\n+test_atexit () {\n+\t# We cannot detect when we are in a subshell in general, but by\n+\t# doing so on Bash is better than nothing (the test will\n+\t# silently pass on other shells).\n+\ttest \"${BASH_SUBSHELL-0}\" = 0 ||\n+\terror \"bug in test script: test_atexit does nothing in a subshell\"\n+\ttest_atexit_cleanup=\"{ $*\n+\t\t} && (exit \\\"\\$eval_ret\\\"); eval_ret=\\$?; $test_atexit_cleanup\"\n+}\n+\n+test_atexit_handler () {\n+\ttest : != \"$test_atexit_cleanup\" || return 0\n+\n+\tsetup_malloc_check\n+\ttest_eval_ \"$test_atexit_cleanup\"\n+\ttest_atexit_cleanup=:\n+\tteardown_malloc_check\n+}\n+\n # Most tests can use the created repository, but some may need to create more.\n # Usage: test_create_repo \n test_create_repo () {\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 7ed0013f6d..6f9c1f5300 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -413,6 +413,7 @@ test_external_has_tap=0\n \n die () {\n \tcode=$?\n+\ttest_atexit_handler || code=$?\n \tif test -n \"$GIT_EXIT_OK\"\n \tthen\n \t\texit $code\n@@ -826,9 +827,12 @@ write_junit_xml_testcase () {\n \tjunit_have_testcase=t\n }\n \n+test_atexit_cleanup=:\n test_done () {\n \tGIT_EXIT_OK=t\n \n+\ttest -n \"$immediate\" || test_atexit_handler\n+\n \tif test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n \tthen\n \t\ttest -n \"$junit_have_testcase\" || {\n-- \ngitgitgadget\n\n"},{"id":"360500","messageId":"3bb226b79b87e30a3e8f6360c7f94d9adf4fe1b7.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 08/13] git-daemon: use `test_atexit` in the tests","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:09Z","receivedAt":"2018-10-15T10:12:13Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis makes use of the just-introduced consistent way to specify that a\nlong-running process needs to be terminated at the end of a test script\nrun.\n\nSigned-off-by: Johannes Schindelin \n---\n t/interop/i5500-git-daemon.sh | 1 -\n t/lib-git-daemon.sh | 3 +--\n t/t5570-git-daemon.sh | 1 -\n 3 files changed, 1 insertion(+), 4 deletions(-)\n\ndiff --git a/t/interop/i5500-git-daemon.sh b/t/interop/i5500-git-daemon.sh\nindex 1daf69420b..4d22e42f84 100755\n--- a/t/interop/i5500-git-daemon.sh\n+++ b/t/interop/i5500-git-daemon.sh\n@@ -37,5 +37,4 @@ test_expect_success \"fetch with $VERSION_B\" '\n \ttest_cmp expect actual\n '\n \n-stop_git_daemon\n test_done\ndiff --git a/t/lib-git-daemon.sh b/t/lib-git-daemon.sh\nindex edbea2d986..a896af2284 100644\n--- a/t/lib-git-daemon.sh\n+++ b/t/lib-git-daemon.sh\n@@ -13,7 +13,6 @@\n #\n #\ttest_expect_success ...\n #\n-#\tstop_git_daemon\n #\ttest_done\n \n test_tristate GIT_TEST_GIT_DAEMON\n@@ -43,7 +42,7 @@ start_git_daemon() {\n \n \tmkdir -p \"$GIT_DAEMON_DOCUMENT_ROOT_PATH\"\n \n-\ttrap 'code=$?; stop_git_daemon; (exit $code); die' EXIT\n+\ttest_atexit 'stop_git_daemon'\n \n \tsay >&3 \"Starting git daemon ...\"\n \tmkfifo git_daemon_output\ndiff --git a/t/t5570-git-daemon.sh b/t/t5570-git-daemon.sh\nindex 7466aad111..08f95c80a2 100755\n--- a/t/t5570-git-daemon.sh\n+++ b/t/t5570-git-daemon.sh\n@@ -211,5 +211,4 @@ test_expect_success FAKENC 'hostname interpolation works after LF-stripping' '\n \ttest_cmp expect actual\n '\n \n-stop_git_daemon\n test_done\n-- \ngitgitgadget\n\n"},{"id":"360501","messageId":"3e2193a73de0b68d5a38f1792642c828f4aed1db.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 09/13] git-p4: use `test_atexit` to kill the daemon","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:11Z","receivedAt":"2018-10-15T10:12:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis should be more reliable than the current method, and prepares the\ntest suite for a consistent way to clean up before re-running the tests\nwith different options.\n\nSigned-off-by: Johannes Schindelin \n---\n t/lib-git-p4.sh | 10 +---------\n t/t0000-basic.sh | 2 ++\n t/t9800-git-p4-basic.sh | 4 ----\n t/t9801-git-p4-branch.sh | 4 ----\n t/t9802-git-p4-filetype.sh | 4 ----\n t/t9803-git-p4-shell-metachars.sh | 4 ----\n t/t9804-git-p4-label.sh | 4 ----\n t/t9805-git-p4-skip-submit-edit.sh | 4 ----\n t/t9806-git-p4-options.sh | 5 -----\n t/t9807-git-p4-submit.sh | 4 ----\n t/t9808-git-p4-chdir.sh | 4 ----\n t/t9809-git-p4-client-view.sh | 4 ----\n t/t9810-git-p4-rcs.sh | 4 ----\n t/t9811-git-p4-label-import.sh | 5 -----\n t/t9812-git-p4-wildcards.sh | 4 ----\n t/t9813-git-p4-preserve-users.sh | 4 ----\n t/t9814-git-p4-rename.sh | 4 ----\n t/t9815-git-p4-submit-fail.sh | 4 ----\n t/t9816-git-p4-locked.sh | 4 ----\n t/t9817-git-p4-exclude.sh | 4 ----\n t/t9818-git-p4-block.sh | 4 ----\n t/t9819-git-p4-case-folding.sh | 4 ----\n t/t9820-git-p4-editor-handling.sh | 4 ----\n t/t9821-git-p4-path-variations.sh | 4 ----\n t/t9822-git-p4-path-encoding.sh | 4 ----\n t/t9823-git-p4-mock-lfs.sh | 4 ----\n t/t9824-git-p4-git-lfs.sh | 4 ----\n t/t9825-git-p4-handle-utf16-without-bom.sh | 4 ----\n t/t9826-git-p4-keep-empty-commits.sh | 4 ----\n t/t9827-git-p4-change-filetype.sh | 4 ----\n t/t9828-git-p4-map-user.sh | 4 ----\n t/t9829-git-p4-jobs.sh | 4 ----\n t/t9830-git-p4-symlink-dir.sh | 4 ----\n t/t9831-git-p4-triggers.sh | 4 ----\n t/t9832-unshelve.sh | 4 ----\n t/t9833-errors.sh | 5 -----\n 36 files changed, 3 insertions(+), 148 deletions(-)\n\ndiff --git a/t/lib-git-p4.sh b/t/lib-git-p4.sh\nindex c27599474c..f4f5d7d296 100644\n--- a/t/lib-git-p4.sh\n+++ b/t/lib-git-p4.sh\n@@ -74,15 +74,6 @@ cli=\"$TRASH_DIRECTORY/cli\"\n git=\"$TRASH_DIRECTORY/git\"\n pidfile=\"$TRASH_DIRECTORY/p4d.pid\"\n \n-# Sometimes \"prove\" seems to hang on exit because p4d is still running\n-cleanup () {\n-\tif test -f \"$pidfile\"\n-\tthen\n-\t\tkill -9 $(cat \"$pidfile\") 2>/dev/null && exit 255\n-\tfi\n-}\n-trap cleanup EXIT\n-\n # git p4 submit generates a temp file, which will\n # not get cleaned up if the submission fails. Don't\n # clutter up /tmp on the test machine.\n@@ -141,6 +132,7 @@ start_p4d () {\n \t\t# p4d failed to start\n \t\treturn 1\n \tfi\n+\ttest_atexit kill_p4d\n \n \t# build a p4 user so author@example.com has an entry\n \tp4_add_user author\ndiff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\nindex 8c5faa6ce1..041bd7e3ce 100755\n--- a/t/t0000-basic.sh\n+++ b/t/t0000-basic.sh\n@@ -134,6 +134,7 @@ check_sub_test_lib_test_err () {\n \t)\n }\n \n+cat >/dev/null <<\\DDD\n test_expect_success 'pretend we have a fully passing test suite' \"\n \trun_sub_test_lib_test full-pass '3 passing tests' <<-\\\\EOF &&\n \tfor i in 1 2 3\n@@ -820,6 +821,7 @@ test_expect_success 'tests clean up even on failures' \"\n \t> 1..2\n \tEOF\n \"\n+DDD\n \n test_expect_success 'test_atexit is run' \"\n \ttest_must_fail run_sub_test_lib_test \\\ndiff --git a/t/t9800-git-p4-basic.sh b/t/t9800-git-p4-basic.sh\nindex 729cd25770..5856563068 100755\n--- a/t/t9800-git-p4-basic.sh\n+++ b/t/t9800-git-p4-basic.sh\n@@ -326,8 +326,4 @@ test_expect_success 'submit from worktree' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9801-git-p4-branch.sh b/t/t9801-git-p4-branch.sh\nindex 6a86d6996b..50013132c8 100755\n--- a/t/t9801-git-p4-branch.sh\n+++ b/t/t9801-git-p4-branch.sh\n@@ -610,8 +610,4 @@ test_expect_success 'Update a file in git side and submit to P4 using client vie\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9802-git-p4-filetype.sh b/t/t9802-git-p4-filetype.sh\nindex 9978352d78..94edebe272 100755\n--- a/t/t9802-git-p4-filetype.sh\n+++ b/t/t9802-git-p4-filetype.sh\n@@ -333,8 +333,4 @@ test_expect_success SYMLINKS 'empty symlink target' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9803-git-p4-shell-metachars.sh b/t/t9803-git-p4-shell-metachars.sh\nindex d5c3675100..2913277013 100755\n--- a/t/t9803-git-p4-shell-metachars.sh\n+++ b/t/t9803-git-p4-shell-metachars.sh\n@@ -105,8 +105,4 @@ test_expect_success 'branch with shell char' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9804-git-p4-label.sh b/t/t9804-git-p4-label.sh\nindex e30f80e617..3236457106 100755\n--- a/t/t9804-git-p4-label.sh\n+++ b/t/t9804-git-p4-label.sh\n@@ -108,8 +108,4 @@ test_expect_failure 'two labels on the same changelist' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9805-git-p4-skip-submit-edit.sh b/t/t9805-git-p4-skip-submit-edit.sh\nindex 5fbf904dc8..90ef647db7 100755\n--- a/t/t9805-git-p4-skip-submit-edit.sh\n+++ b/t/t9805-git-p4-skip-submit-edit.sh\n@@ -98,8 +98,4 @@ test_expect_success 'no config, edited' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9806-git-p4-options.sh b/t/t9806-git-p4-options.sh\nindex 3f5291b857..4e794a01bf 100755\n--- a/t/t9806-git-p4-options.sh\n+++ b/t/t9806-git-p4-options.sh\n@@ -300,9 +300,4 @@ test_expect_success 'use --git-dir option and GIT_DIR' '\n \ttest_path_is_file \"$git\"/cli_file2.t\n '\n \n-\n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9807-git-p4-submit.sh b/t/t9807-git-p4-submit.sh\nindex 2325599ee6..488d916c10 100755\n--- a/t/t9807-git-p4-submit.sh\n+++ b/t/t9807-git-p4-submit.sh\n@@ -542,8 +542,4 @@ test_expect_success 'submit --update-shelve' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9808-git-p4-chdir.sh b/t/t9808-git-p4-chdir.sh\nindex 11d2b5102c..58a9b3b71e 100755\n--- a/t/t9808-git-p4-chdir.sh\n+++ b/t/t9808-git-p4-chdir.sh\n@@ -83,8 +83,4 @@ test_expect_success SYMLINKS 'p4 client root symlink should stay symbolic' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9809-git-p4-client-view.sh b/t/t9809-git-p4-client-view.sh\nindex 897b3c3034..3cff1fce1b 100755\n--- a/t/t9809-git-p4-client-view.sh\n+++ b/t/t9809-git-p4-client-view.sh\n@@ -836,8 +836,4 @@ test_expect_success 'quotes on both sides' '\n \tgit_verify \"cdir 1/file11\" \"cdir 1/file12\"\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9810-git-p4-rcs.sh b/t/t9810-git-p4-rcs.sh\nindex cc53debe19..57b533dc6f 100755\n--- a/t/t9810-git-p4-rcs.sh\n+++ b/t/t9810-git-p4-rcs.sh\n@@ -360,8 +360,4 @@ test_expect_failure 'Add keywords in git which do not match the default p4 value\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9811-git-p4-label-import.sh b/t/t9811-git-p4-label-import.sh\nindex 602b0a5d5c..b70e81c3cd 100755\n--- a/t/t9811-git-p4-label-import.sh\n+++ b/t/t9811-git-p4-label-import.sh\n@@ -259,9 +259,4 @@ test_expect_success 'importing labels with missing revisions' '\n \t)\n '\n \n-\n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9812-git-p4-wildcards.sh b/t/t9812-git-p4-wildcards.sh\nindex 0206771fbb..254a7c2446 100755\n--- a/t/t9812-git-p4-wildcards.sh\n+++ b/t/t9812-git-p4-wildcards.sh\n@@ -211,8 +211,4 @@ test_expect_success 'wildcard files requiring keyword scrub' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9813-git-p4-preserve-users.sh b/t/t9813-git-p4-preserve-users.sh\nindex 783c6ad165..fd018c87a8 100755\n--- a/t/t9813-git-p4-preserve-users.sh\n+++ b/t/t9813-git-p4-preserve-users.sh\n@@ -138,8 +138,4 @@ test_expect_success 'not preserving user with mixed authorship' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9814-git-p4-rename.sh b/t/t9814-git-p4-rename.sh\nindex 60baa06e27..468767cbf4 100755\n--- a/t/t9814-git-p4-rename.sh\n+++ b/t/t9814-git-p4-rename.sh\n@@ -242,8 +242,4 @@ test_expect_success P4D_HAVE_CONFIGURABLE_RUN_MOVE_ALLOW \\\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9815-git-p4-submit-fail.sh b/t/t9815-git-p4-submit-fail.sh\nindex eaf03a6563..9779dc0d11 100755\n--- a/t/t9815-git-p4-submit-fail.sh\n+++ b/t/t9815-git-p4-submit-fail.sh\n@@ -422,8 +422,4 @@ test_expect_success 'cleanup chmod after submit cancel' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9816-git-p4-locked.sh b/t/t9816-git-p4-locked.sh\nindex d048bd33fa..932841003c 100755\n--- a/t/t9816-git-p4-locked.sh\n+++ b/t/t9816-git-p4-locked.sh\n@@ -138,8 +138,4 @@ test_expect_failure 'move with lock taken' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9817-git-p4-exclude.sh b/t/t9817-git-p4-exclude.sh\nindex aac568eadf..96d25f0c02 100755\n--- a/t/t9817-git-p4-exclude.sh\n+++ b/t/t9817-git-p4-exclude.sh\n@@ -64,8 +64,4 @@ test_expect_success 'clone, then sync with exclude' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9818-git-p4-block.sh b/t/t9818-git-p4-block.sh\nindex ce7cb22ad3..0db7ab9918 100755\n--- a/t/t9818-git-p4-block.sh\n+++ b/t/t9818-git-p4-block.sh\n@@ -146,8 +146,4 @@ test_expect_success 'Clone repo with self-sizing block size' '\n \ttest_line_count \\> 10 log\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9819-git-p4-case-folding.sh b/t/t9819-git-p4-case-folding.sh\nindex d808c008c1..600ce1e0b0 100755\n--- a/t/t9819-git-p4-case-folding.sh\n+++ b/t/t9819-git-p4-case-folding.sh\n@@ -53,8 +53,4 @@ test_expect_failure 'Clone UC repo with lc name' '\n \ttest_must_fail git p4 clone //depot/uc/...\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9820-git-p4-editor-handling.sh b/t/t9820-git-p4-editor-handling.sh\nindex 3c22f74bd4..fa1bba1dd9 100755\n--- a/t/t9820-git-p4-editor-handling.sh\n+++ b/t/t9820-git-p4-editor-handling.sh\n@@ -31,8 +31,4 @@ test_expect_success 'EDITOR with options' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9821-git-p4-path-variations.sh b/t/t9821-git-p4-path-variations.sh\nindex 81e46acfa8..ef80f1690b 100755\n--- a/t/t9821-git-p4-path-variations.sh\n+++ b/t/t9821-git-p4-path-variations.sh\n@@ -193,8 +193,4 @@ test_expect_success 'Add a new file and clone path with new file (ignorecase)' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9822-git-p4-path-encoding.sh b/t/t9822-git-p4-path-encoding.sh\nindex c78477c19b..1bf7635016 100755\n--- a/t/t9822-git-p4-path-encoding.sh\n+++ b/t/t9822-git-p4-path-encoding.sh\n@@ -67,8 +67,4 @@ test_expect_success 'Delete iso8859-1 encoded paths and clone' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9823-git-p4-mock-lfs.sh b/t/t9823-git-p4-mock-lfs.sh\nindex 1f2dc369bf..88b76dc4d6 100755\n--- a/t/t9823-git-p4-mock-lfs.sh\n+++ b/t/t9823-git-p4-mock-lfs.sh\n@@ -185,8 +185,4 @@ test_expect_success 'Run git p4 submit in repo configured with large file system\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh\nindex ed80ca858c..a28dbbdd56 100755\n--- a/t/t9824-git-p4-git-lfs.sh\n+++ b/t/t9824-git-p4-git-lfs.sh\n@@ -287,8 +287,4 @@ test_expect_success 'Add big files to repo and store files in LFS based on compr\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9825-git-p4-handle-utf16-without-bom.sh b/t/t9825-git-p4-handle-utf16-without-bom.sh\nindex 1551845dc1..f049ff8229 100755\n--- a/t/t9825-git-p4-handle-utf16-without-bom.sh\n+++ b/t/t9825-git-p4-handle-utf16-without-bom.sh\n@@ -43,8 +43,4 @@ test_expect_failure 'clone depot with invalid UTF-16 file in non-verbose mode' '\n \tgit p4 clone --dest=\"$git\" //depot\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9826-git-p4-keep-empty-commits.sh b/t/t9826-git-p4-keep-empty-commits.sh\nindex fa8b9daf1f..fd64afe064 100755\n--- a/t/t9826-git-p4-keep-empty-commits.sh\n+++ b/t/t9826-git-p4-keep-empty-commits.sh\n@@ -127,8 +127,4 @@ test_expect_success 'Clone repo subdir with all history' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9827-git-p4-change-filetype.sh b/t/t9827-git-p4-change-filetype.sh\nindex 7433998f47..d3670bd7a2 100755\n--- a/t/t9827-git-p4-change-filetype.sh\n+++ b/t/t9827-git-p4-change-filetype.sh\n@@ -59,8 +59,4 @@ test_expect_success SYMLINKS 'change symbolic link to file' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9828-git-p4-map-user.sh b/t/t9828-git-p4-map-user.sh\nindex e20395c89f..ca6c2942bd 100755\n--- a/t/t9828-git-p4-map-user.sh\n+++ b/t/t9828-git-p4-map-user.sh\n@@ -54,8 +54,4 @@ test_expect_success 'Clone repo root path with all history' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9829-git-p4-jobs.sh b/t/t9829-git-p4-jobs.sh\nindex 971aeeea1f..88cfb1fcd3 100755\n--- a/t/t9829-git-p4-jobs.sh\n+++ b/t/t9829-git-p4-jobs.sh\n@@ -92,8 +92,4 @@ test_expect_success 'check log message of changelist with more jobs' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9830-git-p4-symlink-dir.sh b/t/t9830-git-p4-symlink-dir.sh\nindex 2ad1b0810d..3fb6960c18 100755\n--- a/t/t9830-git-p4-symlink-dir.sh\n+++ b/t/t9830-git-p4-symlink-dir.sh\n@@ -36,8 +36,4 @@ test_expect_success 'symlinked directory' '\n \n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9831-git-p4-triggers.sh b/t/t9831-git-p4-triggers.sh\nindex be44c9751a..d743ca33ee 100755\n--- a/t/t9831-git-p4-triggers.sh\n+++ b/t/t9831-git-p4-triggers.sh\n@@ -96,8 +96,4 @@ test_expect_success 'submit description with extra info lines from verbose p4 ch\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9832-unshelve.sh b/t/t9832-unshelve.sh\nindex 48ec7679b8..b649d1b7c3 100755\n--- a/t/t9832-unshelve.sh\n+++ b/t/t9832-unshelve.sh\n@@ -131,8 +131,4 @@ test_expect_success 'try to unshelve the change' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n test_done\ndiff --git a/t/t9833-errors.sh b/t/t9833-errors.sh\nindex 277d347012..1f3d879122 100755\n--- a/t/t9833-errors.sh\n+++ b/t/t9833-errors.sh\n@@ -72,9 +72,4 @@ test_expect_success 'git operation with expired ticket' '\n \t)\n '\n \n-test_expect_success 'kill p4d' '\n-\tkill_p4d\n-'\n-\n-\n test_done\n-- \ngitgitgadget\n\n"},{"id":"360502","messageId":"ae3c42519abff7ef32c767f9587ef7f0160033ed.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 10/13] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:12Z","receivedAt":"2018-10-15T10:12:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe JUnit XML format lends itself to be presented in a powerful UI,\nwhere you can drill down to the information you are interested in very\nquickly.\n\nFor test failures, this usually means that you want to see the detailed\ntrace of the failing tests.\n\nWith Travis CI, we passed the `--verbose-log` option to get those\ntraces. However, that seems excessive, as we do not need/use the logs in\nalmost all of those cases: only when a test fails do we have a way to\ninclude the trace.\n\nSo let's do something different when using Azure DevOps: let's run all\nthe tests with `--quiet` first, and only if a failure is encountered,\ntry to trace the commands as they are executed.\n\nOf course, we cannot turn on `--verbose-log` after the fact. So let's\njust re-run the test with all the same options, adding `--verbose-log`.\nAnd then munging the output file into the JUnit XML on the fly.\n\nNote: there is an off chance that re-running the test in verbose mode\n\"fixes\" the failures (and this does happen from time to time!). That is\na possibility we should be able to live with. Ideally, we would label\nthis as \"Passed upon rerun\", and Azure Pipelines even know about that\noutcome, but it is not available when using the JUnit XML format for\nnow:\nhttps://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 34 +++++++++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 6f9c1f5300..8a60e39364 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -85,6 +85,13 @@ done,*)\n \ttest \"$(cat \"$BASE.exit\")\" = 0\n \texit\n \t;;\n+*' --write-junit-xml '*)\n+\t# record how to call this script *with* --verbose-log, in case\n+\t# we encounter a breakage\n+\tjunit_rerun_options_sq=\"$(printf '%s\\n' \"$0\" --verbose-log -x \"$@\" |\n+\t\tsed -e \"s/'/'\\\\\\\\''/g\" -e \"s/^/'/\" -e \"s/\\$/'/\" |\n+\t\ttr '\\012' ' ')\"\n+\t;;\n esac\n \n # For repeatability, reset the environment to known value.\n@@ -446,10 +453,31 @@ test_ok_ () {\n test_failure_ () {\n \tif test -n \"$write_junit_xml\"\n \tthen\n+\t\tif test -z \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\tthen\n+\t\t\t# clean up\n+\t\t\ttest_atexit_handler\n+\n+\t\t\t# re-run with --verbose-log\n+\t\t\techo \"# Re-running: $junit_rerun_options_sq\" >&2\n+\n+\t\t\tcd \"$TEST_DIRECTORY\" &&\n+\t\t\teval \"${TEST_SHELL_PATH}\" \"$junit_rerun_options_sq\" \\\n+\t\t\t\t>/dev/null 2>&1\n+\t\t\tstatus=$?\n+\n+\t\t\tsay_color \"\" \"$(test 0 = $status ||\n+\t\t\t\techo \"not \")ok $test_count - (re-ran with trace)\"\n+\t\t\tsay \"1..$test_count\"\n+\t\t\tGIT_EXIT_OK=t\n+\t\t\texit $status\n+\t\tfi\n+\n \t\tjunit_insert=\"\"\n \t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n-\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n+\t\t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n \t\tjunit_insert=\"$junit_insert\"\n \t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n \tfi\n@@ -734,6 +762,10 @@ test_start_ () {\n \tif test -n \"$write_junit_xml\"\n \tthen\n \t\tjunit_start=$(test-tool date getnanos)\n+\n+\t\t# truncate output\n+\t\ttest -z \"$GIT_TEST_TEE_OUTPUT_FILE\" ||\n+\t\t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n \tfi\n }\n \n-- \ngitgitgadget\n\n"},{"id":"360503","messageId":"2466a48aa3eedc703263c2b63c9c7b5f13f0194a.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 11/13] tests: record more stderr with --write-junit-xml in case of failure","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:14Z","receivedAt":"2018-10-15T10:12:17Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nSometimes, failures in a test case are actually caused by issues in\nearlier test cases.\n\nTo make it easier to see those issues, let's attach the output from\nbefore the failing test case (i.e. stdout/stderr since the previous\nfailing test case, or the start of the test script). This will be\nvisible in the \"Attachments\" of the details of the failed test.\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 12 +++++++++---\n 1 file changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 8a60e39364..5f62418f9c 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -479,6 +479,9 @@ test_failure_ () {\n \t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n \t\t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n \t\tjunit_insert=\"$junit_insert\"\n+\t\tjunit_insert=\"$junit_insert$(xml_attr_encode \\\n+\t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE.err\")\")\"\n+\t\t>\"$GIT_TEST_TEE_OUTPUT_FILE.err\"\n \t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n \tfi\n \ttest_failure=$(($test_failure + 1))\n@@ -763,9 +766,12 @@ test_start_ () {\n \tthen\n \t\tjunit_start=$(test-tool date getnanos)\n \n-\t\t# truncate output\n-\t\ttest -z \"$GIT_TEST_TEE_OUTPUT_FILE\" ||\n-\t\t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\t# append to future ; truncate output\n+\t\ttest -z \"$GIT_TEST_TEE_OUTPUT_FILE\" || {\n+\t\t\tcat \"$GIT_TEST_TEE_OUTPUT_FILE\" \\\n+\t\t\t\t>>\"$GIT_TEST_TEE_OUTPUT_FILE.err\"\n+\t\t\t>\"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\t}\n \tfi\n }\n \n-- \ngitgitgadget\n\n"},{"id":"360504","messageId":"d112b3fe86e2d5168161cd3edf9616ff7bec3319.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 12/13] README: add a build badge (status of the Azure Pipelines build)","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:15Z","receivedAt":"2018-10-15T10:12:18Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nJust like so many other OSS projects, we now also have a build badge.\n\nSigned-off-by: Johannes Schindelin \n---\n README.md | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/README.md b/README.md\nindex f920a42fad..bf4780c22d 100644\n--- a/README.md\n+++ b/README.md\n@@ -1,3 +1,5 @@\n+[![Build Status](https:/dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2)\n+\n Git - fast, scalable, distributed revision control system\n =========================================================\n \n-- \ngitgitgadget\n\n"},{"id":"360505","messageId":"0a53f371350146ecd01b44f0f42da447d7573689.1539598316.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v2 13/13] travis: fix skipping tagged releases","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2018-10-15T10:12:17Z","receivedAt":"2018-10-15T10:12:21Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWhen building a PR, TRAVIS_BRANCH refers to the *target branch*.\nTherefore, if a PR targets `master`, and `master` happened to be tagged,\nwe skipped the build by mistake.\n\nFix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*)\nwhen available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also\nknown as \"push builds\").\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 584abcd529..e1858ae609 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -3,7 +3,7 @@\n if test true = \"$TRAVIS\"\n then\n \t# We are running within Travis CI\n-\tCI_BRANCH=\"$TRAVIS_BRANCH\"\n+\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n \tCI_COMMIT=\"$TRAVIS_COMMIT\"\n \tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n \tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n-- \ngitgitgadget\n"},{"id":"360511","messageId":"CAE5ih79cNkVVQOkaeJifSqkcp=HGmRb3a=jJC=g08iV8dYRyXg@mail.gmail.com","threadId":"49270","inReplyTo":"3e2193a73de0b68d5a38f1792642c828f4aed1db.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/13] git-p4: use `test_atexit` to kill the daemon","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2018-10-15T11:00:50Z","receivedAt":"2018-10-15T11:01:05Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On Mon, 15 Oct 2018 at 11:12, Johannes Schindelin via GitGitGadget\n wrote:\n>\n> From: Johannes Schindelin \n>\n> This should be more reliable than the current method, and prepares the\n> test suite for a consistent way to clean up before re-running the tests\n> with different options.\n>\n\nI'm finding that it's leaving p4d processes lying around.\n\ne.g.\n\n$ ./t9820-git-p4-editor-handling.sh\n\n$ ./t9820-git-p4-editor-handling.sh\n\n\nAnd also\n\n$ ./t9800-git-p4-basic.sh\n\nCtrl-C\n\n$ ./t9800-git-p4-basic.sh\n\n\n$ ps | grep p4d\n21392 pts/1 00:00:00 p4d <<<<<\n\n\nLuke\n"},{"id":"360524","messageId":"20181015142215.GB1764@syl.hsd1.wa.comcast.net","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-10-15T14:22:15Z","receivedAt":"2018-10-15T14:22:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Johannes,\n\nThanks for putting this together, and offering to build Git on Azure\nPipelines. I haven't followed v1 of this series very closely, so please\nexcuse me if my comments have already been addressed, and I missed them\nin a skim of the last revision.\n\nOn Mon, Oct 15, 2018 at 03:11:57AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> It is also an invaluable tool for contributors who can validate their code\n> contributions via PRs on GitHub, e.g. to verify that their tests do actually\n> run on macOS (i.e. with the BSD family of Unix tools instead of the GNU\n> one).\n\nAgree.\n\n> The one sad part about this is the Windows support. Travis lacks it, and we\n> work around that by using Azure Pipelines (the CI part of Azure DevOps,\n> formerly known as Visual Studio Team Services) indirectly: one phase in\n> Travis would trigger a build, wait for its log, and then paste that log.\n\nI wonder if Travis' recent announcement [1] affects this at all. To\nsummarize [1], Travis is offering an early version of adding Windows to\ntheir list of supported builder operations systems. This brings the list\nto macOS, Linux, and Windows, which I think satisfies what we would\nlike to regularly build git.git on.\n\nWould we like to abandon Travis as our main CI service for upstream\ngit.git, and build on Azure Pipelines only? If so, I think that this is\nan OK way to go, but I think that I would be opposed to having more than\none build system. (FWIW, we tend to _three_ for Git LFS, and it can be a\nhassle at times).\n\nI see some benefit to sticking with Travis, since we already have a\nbuild configuration that works there. But, you've done the work to\n\"port\" that build configuration over to Azure, so perhaps the point is\nmoot.\n\n> As Git's Windows builds (and tests!) take quite a bit of time, Travis often\n> timed out, or somehow the trigger did not work, and for security reasons\n> (the Windows builds are performed in a private pool of containers), the\n> Windows builds are completely disabled for Pull Requests on GitHub.\n\nThis would be a concession of [1], in my mind: is it possible to run the\ntests on Windows in a time such that Travis will not time out?\n\n> As a special treat, this patch series adds the ability to present the\n> outcome of Git's test suite as JUnit-style .xml files. This allows the Azure\n> Pipelines build to present fun diagrams, trends, and makes it a lot easier\n> to drill down to test failures than before. See for example\n> https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details\n> [https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details]\n> (you can click on the label of the failed test, and then see the detailed\n> output in the right pane).\n\nThat's pretty cool. Travis doesn't support this (to the best of my\nknowledge).\n\nThanks,\nTaylor\n\n[1]: https://blog.travis-ci.com/2018-10-11-windows-early-release\n"},{"id":"360529","messageId":"nycvar.QRO.7.76.6.1810151636510.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"20181015142215.GB1764@syl.hsd1.wa.comcast.net","subject":"Re: [PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-15T14:55:25Z","receivedAt":"2018-10-15T14:55:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Taylor,\n\nOn Mon, 15 Oct 2018, Taylor Blau wrote:\n\n> Thanks for putting this together, and offering to build Git on Azure\n> Pipelines. I haven't followed v1 of this series very closely, so please\n> excuse me if my comments have already been addressed, and I missed them\n> in a skim of the last revision.\n> \n> On Mon, Oct 15, 2018 at 03:11:57AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> > It is also an invaluable tool for contributors who can validate their code\n> > contributions via PRs on GitHub, e.g. to verify that their tests do actually\n> > run on macOS (i.e. with the BSD family of Unix tools instead of the GNU\n> > one).\n> \n> Agree.\n> \n> > The one sad part about this is the Windows support. Travis lacks it, and we\n> > work around that by using Azure Pipelines (the CI part of Azure DevOps,\n> > formerly known as Visual Studio Team Services) indirectly: one phase in\n> > Travis would trigger a build, wait for its log, and then paste that log.\n> \n> I wonder if Travis' recent announcement [1] affects this at all.\n\n:-)\n\nIt did not escape my notice that after years and years of trying to get\n*anybody* at Travis to listen to my offers to help get this started, the\nannouncement of Azure Pipelines for OSS seemed to finally do the trick\n(they still don't bother to talk to me, of course).\n\nAnd to answer your question without a question mark: I do not really think\nthat the Travis announcement affects this here patch series: I have a ton\nof good experience with Azure Pipelines, use it in Git for Windows for\nages, and I am finally able to have it in core Git, too. So I want it, and\nI spent a lot of time getting there, and I think it probably won't hurt\ncore Git at all (besides, it seems that at least some of the phases are a\nbit faster on Azure Pipelines than Travis).\n\nAnother really good reason for me to do this is that I can prod the Azure\nPipelines team directly. And I even get an answer, usually within minutes.\nWhich is a lot faster than the Travis team answers my questions, which\nis... not yet? (I tried to get in contact with them in late 2015 or early\n2016, and I tried again a year later, and then a couple of months later,\nand I have yet to hear back.)\n\nAlso, I am not quite sure about the timeouts on Travis, but at least\nAppVeyor had rather short timeouts: the Windows build (due to our\nextensive use of Unix shell scripting in our test suite) takes 1h40m\ncurrently, and AppVeyor times out after 20 or 30 minutes. I could imagine\nthat Travis times out after the same time, or maybe 60 minutes, which\nwould still be too short. On Azure Pipelines, the maximum timeout (which\ncan be configured via the azure-pipelines.yml file) is four hours IIRC.\nPlenty enough even for our test suite on Windows.\n\n> To summarize [1], Travis is offering an early version of adding Windows\n> to their list of supported builder operations systems. This brings the\n> list to macOS, Linux, and Windows, which I think satisfies what we would\n> like to regularly build git.git on.\n\nHonestly, I would love to have also FreeBSD and other platforms being\ntested. And with Azure Pipelines, I can make that happen (eventually), by\nadding another pool of VMs (given that I have a free $150/month Azure\nsubscription, I'd use Azure VMs, of course). As long as a platform can run\n.NET Core software, it can run Azure Pipelines agents.\n\nWith Travis, I don't think I can add private agent pools.\n\n> Would we like to abandon Travis as our main CI service for upstream\n> git.git, and build on Azure Pipelines only? If so, I think that this is\n> an OK way to go, but I think that I would be opposed to having more than\n> one build system. (FWIW, we tend to _three_ for Git LFS, and it can be a\n> hassle at times).\n\nThis question of abandoning Travis in favor of Azure Pipelines is a bit of\na hornets' nest, as I really, really only want to bring the goodness of\nAzure Pipelines to git.git, and I am *clearly* biased, as I work at\nMicrosoft.\n\nWhich is the reason why I did not even hint at it in the cover letter, let\nalone included a patch to make it so.\n\nMy patch series is purely about adding support for running CI/PR builds of\nhttps://github.com/git/git via Azure Pipelines.\n\n> I see some benefit to sticking with Travis, since we already have a\n> build configuration that works there. But, you've done the work to\n> \"port\" that build configuration over to Azure, so perhaps the point is\n> moot.\n\nIt is not so much a port, as an attempt to generalize our ci/* files.\n\n> > As Git's Windows builds (and tests!) take quite a bit of time, Travis often\n> > timed out, or somehow the trigger did not work, and for security reasons\n> > (the Windows builds are performed in a private pool of containers), the\n> > Windows builds are completely disabled for Pull Requests on GitHub.\n> \n> This would be a concession of [1], in my mind: is it possible to run the\n> tests on Windows in a time such that Travis will not time out?\n\nTo be honest, I spent such a lot of time to get things to work on Azure\nPipelines, *and* we get a nice view on the test failures there, too (which\nTravis will probably also offer soon, in response to what Azure Pipelines\noffer ;-)), I cannot really justify spending time on trying to make things\nwork on Travis' Windows VMs, too. Especially when I have to expect to run\ninto timeout issues anyway.\n\n> > As a special treat, this patch series adds the ability to present the\n> > outcome of Git's test suite as JUnit-style .xml files. This allows the Azure\n> > Pipelines build to present fun diagrams, trends, and makes it a lot easier\n> > to drill down to test failures than before. See for example\n> > https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details\n> > [https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details]\n> > (you can click on the label of the failed test, and then see the detailed\n> > output in the right pane).\n> \n> That's pretty cool. Travis doesn't support this (to the best of my\n> knowledge).\n\nExactly.\n\nPlus, if things don't work in Azure Pipelines, I (or one of the other\nMicrosoft employees among the core Git developers) can easily take a\nshortcut to the team and get things fixed. In my mind, that counts for a\nlot, too, especially given my own, frustrating personal experience with\nTravis.\n\nCiao,\nDscho\n\n> [1]: https://blog.travis-ci.com/2018-10-11-windows-early-release\n"},{"id":"360532","messageId":"nycvar.QRO.7.76.6.1810151657080.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAE5ih79cNkVVQOkaeJifSqkcp=HGmRb3a=jJC=g08iV8dYRyXg@mail.gmail.com","subject":"Re: [PATCH v2 09/13] git-p4: use `test_atexit` to kill the daemon","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-15T15:02:24Z","receivedAt":"2018-10-15T15:02:29Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Luke,\n\nOn Mon, 15 Oct 2018, Luke Diamand wrote:\n\n> On Mon, 15 Oct 2018 at 11:12, Johannes Schindelin via GitGitGadget\n> wrote:\n> >\n> > From: Johannes Schindelin \n> >\n> > This should be more reliable than the current method, and prepares the\n> > test suite for a consistent way to clean up before re-running the tests\n> > with different options.\n> >\n> \n> I'm finding that it's leaving p4d processes lying around.\n\nThat's a bummer!\n\n> e.g.\n> \n> $ ./t9820-git-p4-editor-handling.sh\n> \n> $ ./t9820-git-p4-editor-handling.sh\n> \n\nSince I do not currently have a setup with p4d installed, can you run that\nwith `sh -x ...` and see whether this code path is hit?\n\n test_done () {\n GIT_EXIT_OK=t\n\n+ test -n \"$immediate\" || test_atexit_handler\n+\n if test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n then\n\n> And also\n> \n> $ ./t9800-git-p4-basic.sh\n> \n> Ctrl-C\n\nOh, you're right. I think I need to do something in this line:\n\n\ttrap 'exit $?' INT\n\nin t/test-lib.sh, something like\n\n\ttrap 'exit_code=$?; test_atexit_handler; exit $exit_code' INT\n\nwould you agree? (And: could you test?)\n\nThanks,\nDscho\n\n> $ ./t9800-git-p4-basic.sh\n> \n> \n> $ ps | grep p4d\n> 21392 pts/1 00:00:00 p4d <<<<<\n> \n> \n> Luke\n> \n"},{"id":"360533","messageId":"8736t7dzan.fsf@evledraar.gmail.com","threadId":"49270","inReplyTo":"pull.31.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-15T15:06:56Z","receivedAt":"2018-10-15T15:07:01Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Sep 03 2018, Johannes Schindelin via GitGitGadget wrote:\n\n> As a special treat, this patch series adds the ability to present the\n> outcome of Git's test suite as JUnit-style .xml files. This allows the VSTS\n> build to present fun diagrams, trends, and makes it a lot easier to drill\n> down to test failures than before. See for example\n> https://git.visualstudio.com/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details\n> [https://git.visualstudio.com/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details]\n> (you can click on the label of the failed test, and then see the detailed\n> output in the right pane).\n>\n> This patch series took way more time than I had originally planned, but I\n> think that in particular the advanced display of the test results was worth\n> it. Please let me know what you think about this.\n\nI have not reviewed this in any detail, but incorporating this in some\nform or other seems like a no-brainer to me.\n\nIf we have \"free\" (from the perspective of the project) CPU being\noffered by various CI setups let's use it.\n\nIf people don't like it they don't have to look at those CI runs, and\nthe burden of carrying another CI in git.git is minimal, and actually\nthis patch series makes it much easier to incorporate other CI's later,\nsince you took care of moving the Travis-specific bits out.\n\nAs an aside I poked Junio via private mail in late August to see if he'd\nbe interested in pushing to gitlab.com/git/git.git too as part of his\nnormal push-outs. One neat thing that would buy us is the ability to\nhave a .gitlab-ci.yml in git.git and use their CI implementation.\n"},{"id":"360538","messageId":"CACsJy8DeA0Zko4+ZM5F2L6YO5rpvL9LX3H9NB_bVCyO2mJFUzg@mail.gmail.com","threadId":"49270","inReplyTo":"8736t7dzan.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-10-15T15:42:49Z","receivedAt":"2018-10-15T15:43:17Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Oct 15, 2018 at 5:08 PM Ævar Arnfjörð Bjarmason\n wrote:\n> As an aside I poked Junio via private mail in late August to see if he'd\n> be interested in pushing to gitlab.com/git/git.git too as part of his\n> normal push-outs. One neat thing that would buy us is the ability to\n> have a .gitlab-ci.yml in git.git and use their CI implementation.\n\nThat would be great since it allows gitlab forks to have CI support\nfrom the beginning. But gitlab ci has time execution limits if I\nremember correctly and I'm not sure if we'll use it all up before the\nend of the month (or whatever period that is), or do they offer\nsomething special to git.git?\n-- \nDuy\n"},{"id":"360540","messageId":"CAP8UFD3cpXpyx-AtTybPirzduFR7TfhFf2woa-_CU46DN3RaXw@mail.gmail.com","threadId":"49270","inReplyTo":"CACsJy8DeA0Zko4+ZM5F2L6YO5rpvL9LX3H9NB_bVCyO2mJFUzg@mail.gmail.com","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-10-15T16:17:55Z","receivedAt":"2018-10-15T16:18:09Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Oct 15, 2018 at 5:46 PM Duy Nguyen wrote:\n>\n> On Mon, Oct 15, 2018 at 5:08 PM Ævar Arnfjörð Bjarmason\n> wrote:\n> > As an aside I poked Junio via private mail in late August to see if he'd\n> > be interested in pushing to gitlab.com/git/git.git too as part of his\n> > normal push-outs.\n\nPeople at GitLab also have wondered if it would be possible for GitLab\nto host an official Git mirror, and I think it would be nice indeed if\nJunio could push to gitlab.com.\n\n> > One neat thing that would buy us is the ability to\n> > have a .gitlab-ci.yml in git.git and use their CI implementation.\n\nYeah, that would be nice for people who prefer to use GitLab CI.\n\nIn general I agree that it is a good idea to have support for many CI\ntools even if it is not always up-to-date and even if it's not used by\nmany people.\n\n> That would be great since it allows gitlab forks to have CI support\n> from the beginning. But gitlab ci has time execution limits if I\n> remember correctly and I'm not sure if we'll use it all up before the\n> end of the month (or whatever period that is), or do they offer\n> something special to git.git?\n\nI can ask for more time if that would help.\n"},{"id":"360549","messageId":"nycvar.QRO.7.76.6.1810152032140.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAP8UFD3cpXpyx-AtTybPirzduFR7TfhFf2woa-_CU46DN3RaXw@mail.gmail.com","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-15T18:33:20Z","receivedAt":"2018-10-15T18:33:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi team,\n\nOn Mon, 15 Oct 2018, Christian Couder wrote:\n\n> On Mon, Oct 15, 2018 at 5:46 PM Duy Nguyen wrote:\n> >\n> > On Mon, Oct 15, 2018 at 5:08 PM Ævar Arnfjörð Bjarmason\n> > wrote:\n> > > As an aside I poked Junio via private mail in late August to see if he'd\n> > > be interested in pushing to gitlab.com/git/git.git too as part of his\n> > > normal push-outs.\n> \n> People at GitLab also have wondered if it would be possible for GitLab\n> to host an official Git mirror, and I think it would be nice indeed if\n> Junio could push to gitlab.com.\n\nAFAIR Junio does not push to github.com/git/git, it is an automatic\nmirror.\n\nGitLab could easily do the same.\n\nCiao,\nJohannes"},{"id":"360552","messageId":"CAE5ih78gCgJgeXasdm6j+hO8Bk+_zk_-o5FqkMTr-q7fKDej9Q@mail.gmail.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1810151657080.4546@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 09/13] git-p4: use `test_atexit` to kill the daemon","fromName":"Luke Diamand","fromEmail":"luke@diamand.org","sentAt":"2018-10-15T20:19:17Z","receivedAt":"2018-10-15T20:19:33Z","isPatch":true,"sender":{"key":"luke@diamand.org","avatar":"https://avatars.githubusercontent.com/u/5330967?v=4"},"body":"On Mon, 15 Oct 2018 at 16:02, Johannes Schindelin\n wrote:\n>\n> Hi Luke,\n>\n> On Mon, 15 Oct 2018, Luke Diamand wrote:\n>\n> > On Mon, 15 Oct 2018 at 11:12, Johannes Schindelin via GitGitGadget\n> > wrote:\n> > >\n> > > From: Johannes Schindelin \n> > >\n> > > This should be more reliable than the current method, and prepares the\n> > > test suite for a consistent way to clean up before re-running the tests\n> > > with different options.\n> > >\n> >\n> > I'm finding that it's leaving p4d processes lying around.\n>\n> That's a bummer!\n>\n> > e.g.\n> >\n> > $ ./t9820-git-p4-editor-handling.sh\n> > \n> > $ ./t9820-git-p4-editor-handling.sh\n> > \n>\n> Since I do not currently have a setup with p4d installed, can you run that\n> with `sh -x ...` and see whether this code path is hit?\n\nAll you need to do is to put p4 and p4d in your PATH.\n\nhttps://www.perforce.com/downloads/helix-core-p4d\nhttps://www.perforce.com/downloads/helix-command-line-client-p4\n\nThe server is free to use for a small number of users, you don't need\nto do anything to make it go.\n\n\n>\n> test_done () {\n> GIT_EXIT_OK=t\n>\n> + test -n \"$immediate\" || test_atexit_handler\n> +\n\n+ test -n\n+ test_atexit_handler\n./t9820-git-p4-editor-handling.sh: 764:\n./t9820-git-p4-editor-handling.sh: test_atexit_handler: not found\n\nIs that expected?\n\n\n\n\n> if test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n> then\n>\n> > And also\n> >\n> > $ ./t9800-git-p4-basic.sh\n> > \n> > Ctrl-C\n>\n> Oh, you're right. I think I need to do something in this line:\n>\n> trap 'exit $?' INT\n>\n> in t/test-lib.sh, something like\n>\n> trap 'exit_code=$?; test_atexit_handler; exit $exit_code' INT\n>\n> would you agree? (And: could you test?)\n\nNot sure.\nSend me a patch and I can try it out.\n\nThanks,\nLuke\n"},{"id":"360561","messageId":"CAP8UFD0SAwHMfRN8a=LsbChQhHC=Lsi5t6yvmQ4FuxjDXZtSTA@mail.gmail.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1810152032140.4546@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-10-16T00:24:10Z","receivedAt":"2018-10-16T00:24:25Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Mon, Oct 15, 2018 at 8:33 PM Johannes Schindelin\n wrote:\n>\n> Hi team,\n>\n> On Mon, 15 Oct 2018, Christian Couder wrote:\n>\n> > On Mon, Oct 15, 2018 at 5:46 PM Duy Nguyen wrote:\n> > >\n> > > On Mon, Oct 15, 2018 at 5:08 PM Ævar Arnfjörð Bjarmason\n> > > wrote:\n> > > > As an aside I poked Junio via private mail in late August to see if he'd\n> > > > be interested in pushing to gitlab.com/git/git.git too as part of his\n> > > > normal push-outs.\n> >\n> > People at GitLab also have wondered if it would be possible for GitLab\n> > to host an official Git mirror, and I think it would be nice indeed if\n> > Junio could push to gitlab.com.\n>\n> AFAIR Junio does not push to github.com/git/git, it is an automatic\n> mirror.\n\nOn https://git-blame.blogspot.com/p/git-public-repositories.html there is:\n\n\"Copies of the source code to Git live in many repositories, and this\nis a list of the ones I push into. Some repositories have only a\nsubset of branches.\n\nWith maint, master, next, pu, todo:\n\n[...]\nhttps://github.com/git/git/\n\"\n\n> GitLab could easily do the same.\n\nYeah, GitLab could easily create an automatic mirror. I think it would\nbe a bit better though if the git repo on GitLab could be independent.\nIt could make things more resilient against outages or security\nissues.\n"},{"id":"360569","messageId":"20181016025328.GC90754@syl","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1810151636510.4546@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2018-10-16T02:53:28Z","receivedAt":"2018-10-16T02:53:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 15, 2018 at 04:55:25PM +0200, Johannes Schindelin wrote:\n> Hi Taylor,\n>\n> On Mon, 15 Oct 2018, Taylor Blau wrote:\n>\n> > Thanks for putting this together, and offering to build Git on Azure\n> > Pipelines. I haven't followed v1 of this series very closely, so please\n> > excuse me if my comments have already been addressed, and I missed them\n> > in a skim of the last revision.\n> >\n> > On Mon, Oct 15, 2018 at 03:11:57AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> > > It is also an invaluable tool for contributors who can validate their code\n> > > contributions via PRs on GitHub, e.g. to verify that their tests do actually\n> > > run on macOS (i.e. with the BSD family of Unix tools instead of the GNU\n> > > one).\n> >\n> > Agree.\n> >\n> > > The one sad part about this is the Windows support. Travis lacks it, and we\n> > > work around that by using Azure Pipelines (the CI part of Azure DevOps,\n> > > formerly known as Visual Studio Team Services) indirectly: one phase in\n> > > Travis would trigger a build, wait for its log, and then paste that log.\n> >\n> > I wonder if Travis' recent announcement [1] affects this at all.\n>\n> :-)\n>\n> It did not escape my notice that after years and years of trying to get\n> *anybody* at Travis to listen to my offers to help get this started, the\n> announcement of Azure Pipelines for OSS seemed to finally do the trick\n> (they still don't bother to talk to me, of course).\n>\n> And to answer your question without a question mark: I do not really think\n> that the Travis announcement affects this here patch series: I have a ton\n> of good experience with Azure Pipelines, use it in Git for Windows for\n> ages, and I am finally able to have it in core Git, too. So I want it, and\n> I spent a lot of time getting there, and I think it probably won't hurt\n> core Git at all (besides, it seems that at least some of the phases are a\n> bit faster on Azure Pipelines than Travis).\n\nI think that there are fair reasons to prefer Azure Pipelines over\nTravis. In particular, I am encouraged by the fact that we (1) know that\nwe won't timeout, and (2) can have a standardized CI interface on Git\nand Git for Windows. Certainly, the Windows support on Azure Pipelines\nis more developed than that of Travis', so that's another point for\nAzure.\n\n> Another really good reason for me to do this is that I can prod the Azure\n> Pipelines team directly. And I even get an answer, usually within minutes.\n> Which is a lot faster than the Travis team answers my questions, which\n> is... not yet? (I tried to get in contact with them in late 2015 or early\n> 2016, and I tried again a year later, and then a couple of months later,\n> and I have yet to hear back.)\n\nCertainly a good reason. To be clear/fair, I've sent in a number of\nsupport tickets to Travis CI over the years, and have always been\nresponded to in a short amount of time with helpful answers. I think\nthat we would really be fine in either case, TBH.\n\n> Also, I am not quite sure about the timeouts on Travis, but at least\n> AppVeyor had rather short timeouts: the Windows build (due to our\n> extensive use of Unix shell scripting in our test suite) takes 1h40m\n> currently, and AppVeyor times out after 20 or 30 minutes. I could imagine\n> that Travis times out after the same time, or maybe 60 minutes, which\n> would still be too short. On Azure Pipelines, the maximum timeout (which\n> can be configured via the azure-pipelines.yml file) is four hours IIRC.\n> Plenty enough even for our test suite on Windows.\n>\n> > To summarize [1], Travis is offering an early version of adding Windows\n> > to their list of supported builder operations systems. This brings the\n> > list to macOS, Linux, and Windows, which I think satisfies what we would\n> > like to regularly build git.git on.\n>\n> Honestly, I would love to have also FreeBSD and other platforms being\n> tested. And with Azure Pipelines, I can make that happen (eventually), by\n> adding another pool of VMs (given that I have a free $150/month Azure\n> subscription, I'd use Azure VMs, of course). As long as a platform can run\n> .NET Core software, it can run Azure Pipelines agents.\n>\n> With Travis, I don't think I can add private agent pools.\n\nI think that's right.\n\n> > Would we like to abandon Travis as our main CI service for upstream\n> > git.git, and build on Azure Pipelines only? If so, I think that this is\n> > an OK way to go, but I think that I would be opposed to having more than\n> > one build system. (FWIW, we tend to _three_ for Git LFS, and it can be a\n> > hassle at times).\n>\n> This question of abandoning Travis in favor of Azure Pipelines is a bit of\n> a hornets' nest, as I really, really only want to bring the goodness of\n> Azure Pipelines to git.git, and I am *clearly* biased, as I work at\n> Microsoft.\n>\n> Which is the reason why I did not even hint at it in the cover letter, let\n> alone included a patch to make it so.\n>\n> My patch series is purely about adding support for running CI/PR builds of\n> https://github.com/git/git via Azure Pipelines.\n\nI think that adding support for one CI system does carry the weight of\nremoving the other, for the sake of having few CI systems running at any\ngiven time. In other words, even if you don't intend to start a\ndiscussion about removing Travis, those that want run the smallest\nnumber of services (including myself) will ask you about it ;-).\n\nBut I don't see a benefit to dragging this series through the mud by\nspending too much time talking about Travis' future. I think that your\ncover letter does a fine job to address the point, and we can revisit it\nin the future if more people than just myself are opposed to running >1\nCI service.\n\n> > I see some benefit to sticking with Travis, since we already have a\n> > build configuration that works there. But, you've done the work to\n> > \"port\" that build configuration over to Azure, so perhaps the point is\n> > moot.\n>\n> It is not so much a port, as an attempt to generalize our ci/* files.\n>\n> > > As Git's Windows builds (and tests!) take quite a bit of time, Travis often\n> > > timed out, or somehow the trigger did not work, and for security reasons\n> > > (the Windows builds are performed in a private pool of containers), the\n> > > Windows builds are completely disabled for Pull Requests on GitHub.\n> >\n> > This would be a concession of [1], in my mind: is it possible to run the\n> > tests on Windows in a time such that Travis will not time out?\n>\n> To be honest, I spent such a lot of time to get things to work on Azure\n> Pipelines, *and* we get a nice view on the test failures there, too (which\n> Travis will probably also offer soon, in response to what Azure Pipelines\n> offer ;-)), I cannot really justify spending time on trying to make things\n> work on Travis' Windows VMs, too. Especially when I have to expect to run\n> into timeout issues anyway.\n\nAgreed.\n\n> > > As a special treat, this patch series adds the ability to present the\n> > > outcome of Git's test suite as JUnit-style .xml files. This allows the Azure\n> > > Pipelines build to present fun diagrams, trends, and makes it a lot easier\n> > > to drill down to test failures than before. See for example\n> > > https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details\n> > > [https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details]\n> > > (you can click on the label of the failed test, and then see the detailed\n> > > output in the right pane).\n> >\n> > That's pretty cool. Travis doesn't support this (to the best of my\n> > knowledge).\n>\n> Exactly.\n>\n> Plus, if things don't work in Azure Pipelines, I (or one of the other\n> Microsoft employees among the core Git developers) can easily take a\n> shortcut to the team and get things fixed. In my mind, that counts for a\n> lot, too, especially given my own, frustrating personal experience with\n> Travis.\n\nAgreed, and thanks for your response.\n\n> Ciao,\n> Dscho\n>\n> > [1]: https://blog.travis-ci.com/2018-10-11-windows-early-release\n\nThanks,\nTaylor\n"},{"id":"360570","messageId":"CAPig+cQp1jpcQGjvMkCAypQZO_95R5yQDT8Fm=Du2pBOe4Pgbw@mail.gmail.com","threadId":"49270","inReplyTo":"3e2193a73de0b68d5a38f1792642c828f4aed1db.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/13] git-p4: use `test_atexit` to kill the daemon","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-10-16T03:34:53Z","receivedAt":"2018-10-16T03:35:07Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Oct 15, 2018 at 6:12 AM Johannes Schindelin via GitGitGadget\n wrote:\n> This should be more reliable than the current method, and prepares the\n> test suite for a consistent way to clean up before re-running the tests\n> with different options.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\n> @@ -134,6 +134,7 @@ check_sub_test_lib_test_err () {\n> +cat >/dev/null <<\\DDD\n> test_expect_success 'pretend we have a fully passing test suite' \"\n> run_sub_test_lib_test full-pass '3 passing tests' <<-\\\\EOF &&\n> for i in 1 2 3\n> @@ -820,6 +821,7 @@ test_expect_success 'tests clean up even on failures' \"\n> > 1..2\n> EOF\n> \"\n> +DDD\n\nIs this \"DDD\" here-doc leftover debugging goop?\n"},{"id":"360572","messageId":"CAPig+cSf9byf_T696_dzgQLE2X0-61PNXFxX65Ff_RZBGEuD+w@mail.gmail.com","threadId":"49270","inReplyTo":"d112b3fe86e2d5168161cd3edf9616ff7bec3319.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 12/13] README: add a build badge (status of the Azure Pipelines build)","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2018-10-16T03:44:20Z","receivedAt":"2018-10-16T03:44:33Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Oct 15, 2018 at 6:12 AM Johannes Schindelin via GitGitGadget\n wrote:\n> Just like so many other OSS projects, we now also have a build badge.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> diff --git a/README.md b/README.md\n> @@ -1,3 +1,5 @@\n> +[![Build Status](https:/dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2)\n\nThe first URL is broken \"https:/...\" rather than \"https://...\".\n"},{"id":"360578","messageId":"xmqqh8hm1orc.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"8736t7dzan.fsf@evledraar.gmail.com","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-16T04:47:35Z","receivedAt":"2018-10-16T04:47:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> I have not reviewed this in any detail, but incorporating this in some\n> form or other seems like a no-brainer to me.\n>\n> If we have \"free\" (from the perspective of the project) CPU being\n> offered by various CI setups let's use it.\n\nSomebody else said in a separate subthread that there are people who\nwant to limit the CI, and if we pick this we may want to retire\nTravis. Naively I'd say the more the merrier (i.e. agreeing with\nyour above statement), but I may be missing something.\n"},{"id":"360579","messageId":"xmqqd0sa1omm.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1810152032140.4546@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-16T04:50:25Z","receivedAt":"2018-10-16T04:50:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> AFAIR Junio does not push to github.com/git/git, it is an automatic\n> mirror.\n>\n> GitLab could easily do the same.\n\nIt used to be in the early days but these days git/git and\ngitster/git are updated in a same for loop that pushes to various\ndestinations. You are correct that GitLab or any other hosting\nsites could do the same polling and mirroring. I am just too lazy\nto open a new account at yet another hosting site to add that for\nloop, but I may choose to when I am absolutely bored and nothing\nelse to do ;-).\n\n"},{"id":"360582","messageId":"xmqqwoqizclt.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"c9631845104dd66e1be6df9f15eec2481089c85f.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/13] ci: rename the library of common functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-16T05:26:06Z","receivedAt":"2018-10-16T05:26:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> The name is hard-coded to reflect that we use Travis CI for continuous\n> testing.\n>\n> In the next commits, we will extend this to be able use Azure DevOps,\n> too.\n>\n> So let's adjust the name to make it more generic.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n\nWith many pieces still shared across these two CI systems, this\nlooks like an excellent move.\n\nI was wondering if the other extreme of moving ci/* to ci/travis/*,\nintroducing ci/azure/* and sharing nothing was needed, but being\nable to share meaning amount of set-up sounds quite good.\n"},{"id":"360583","messageId":"xmqqsh16zc5x.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"1a22efe849d6da79f2c639c62a1483361a130238.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/13] Add a build definition for Azure DevOps","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-16T05:35:38Z","receivedAt":"2018-10-16T05:35:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> Also, we make use of the shiny new feature we just introduced where the\n> test suite can output JUnit-style .xml files. This information is made\n> available in a nice UI that allows the viewer to filter by phase and/or\n> test number, and to see trends such as: number of (failing) tests, time\n> spent running the test suite, etc.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> azure-pipelines.yml | 319 ++++++++++++++++++++++++++++++++++++++++++\n> ci/mount-fileshare.sh | 26 ++++\n> 2 files changed, 345 insertions(+)\n> create mode 100644 azure-pipelines.yml\n> create mode 100755 ci/mount-fileshare.sh\n\nI wonder if there is a need to keep what is tested by this and\nTravis in sync in any way, but most of the logic is not defined in\nthese \"steps\" but implemented in ci/*.sh scripts to be shared, so it\nwould be OK, I guess.\n\n> diff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\n> new file mode 100755\n> index 0000000000..5fb5f74b70\n> --- /dev/null\n> +++ b/ci/mount-fileshare.sh\n> @@ -0,0 +1,26 @@\n> +#!/bin/sh\n> +\n> +die () {\n> +\techo \"$*\" >&2\n> +\texit 1\n> +}\n> +\n> +test $# = 4 ||\n> +die \"Usage: $0 '.\n\n> +\n> +mkdir -p \"$4\" || die \"Could not create $4\"\n> +\n> +case \"$(uname -s)\" in\n> +Linux)\n> +\tsudo mount -t cifs -o vers=3.0,username=\"$2\",password=\"$3\",dir_mode=0777,file_mode=0777,serverino \"$1\" \"$4\"\n> +\t;;\n> +Darwin)\n> +\tpass=\"$(echo \"$3\" | sed -e 's/\\//%2F/g' -e 's/+/%2B/g')\" &&\n> +\tmount -t smbfs,soft \"smb://$2:$pass@${1#//}\" \"$4\"\n> +\t;;\n> +*)\n> +\tdie \"No support for $(uname -s)\"\n> +\t;;\n> +esac ||\n> +die \"Could not mount $4\"\n> +\n\nTrailing blank line.\n\nThanks.\n"},{"id":"360598","messageId":"nycvar.QRO.7.76.6.1810161047510.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAPig+cSf9byf_T696_dzgQLE2X0-61PNXFxX65Ff_RZBGEuD+w@mail.gmail.com","subject":"Re: [PATCH v2 12/13] README: add a build badge (status of the Azure Pipelines build)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-16T08:48:10Z","receivedAt":"2018-10-16T08:48:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Eric,\n\n\nOn Mon, 15 Oct 2018, Eric Sunshine wrote:\n\n> On Mon, Oct 15, 2018 at 6:12 AM Johannes Schindelin via GitGitGadget\n> wrote:\n> > Just like so many other OSS projects, we now also have a build badge.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > diff --git a/README.md b/README.md\n> > @@ -1,3 +1,5 @@\n> > +[![Build Status](https:/dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2)\n> \n> The first URL is broken \"https:/...\" rather than \"https://...\".\n\nI *swear* I fixed this already. Darn.\n\nThank you,\nDscho\n"},{"id":"360599","messageId":"nycvar.QRO.7.76.6.1810161049100.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAPig+cQp1jpcQGjvMkCAypQZO_95R5yQDT8Fm=Du2pBOe4Pgbw@mail.gmail.com","subject":"Re: [PATCH v2 09/13] git-p4: use `test_atexit` to kill the daemon","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-16T08:51:11Z","receivedAt":"2018-10-16T08:56:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Eric,\n\nOn Mon, 15 Oct 2018, Eric Sunshine wrote:\n\n> On Mon, Oct 15, 2018 at 6:12 AM Johannes Schindelin via GitGitGadget\n> wrote:\n> > This should be more reliable than the current method, and prepares the\n> > test suite for a consistent way to clean up before re-running the tests\n> > with different options.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh\n> > @@ -134,6 +134,7 @@ check_sub_test_lib_test_err () {\n> > +cat >/dev/null <<\\DDD\n> > test_expect_success 'pretend we have a fully passing test suite' \"\n> > run_sub_test_lib_test full-pass '3 passing tests' <<-\\\\EOF &&\n> > for i in 1 2 3\n> > @@ -820,6 +821,7 @@ test_expect_success 'tests clean up even on failures' \"\n> > > 1..2\n> > EOF\n> > \"\n> > +DDD\n> \n> Is this \"DDD\" here-doc leftover debugging goop?\n\nOy, oy, oy. This is definitely a left-over from debugging (as you can\nimagine, it is pretty slow to run t0000-init.sh on Windows, and if I add a\ntest case, the development cycle is much faster with the trick you see\naboive). This left-over even made it into Git for Windows' `master`\nbranch! (Which is the reason I missed it before contributing v2).\n\nWill fix,\nDscho\n"},{"id":"360601","messageId":"CACBZZX59vT1CTiUPfPZDJUSRH=NSzmWiabux=KRdTUFa1R=5bw@mail.gmail.com","threadId":"49270","inReplyTo":"xmqqd0sa1omm.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-16T09:14:06Z","receivedAt":"2018-10-16T09:14:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Tue, Oct 16, 2018 at 6:50 AM Junio C Hamano wrote:\n>\n> Johannes Schindelin writes:\n>\n> > AFAIR Junio does not push to github.com/git/git, it is an automatic\n> > mirror.\n> >\n> > GitLab could easily do the same.\n>\n> It used to be in the early days but these days git/git and\n> gitster/git are updated in a same for loop that pushes to various\n> destinations. You are correct that GitLab or any other hosting\n> sites could do the same polling and mirroring. I am just too lazy\n> to open a new account at yet another hosting site to add that for\n> loop, but I may choose to when I am absolutely bored and nothing\n> else to do ;-).\n\nDo you mind if I squat gitlab.com/git/git in the meantime (i.e. create\nan org etc.) and have it mirror github.com/git/git?, I'll hand the\nkeys over to you if/when you'd like to update your for-loop :)\n\nIn the meantime sometimes GitLab's UI s better than GitHub's, or the\nother way around, so it would be useful to have it there as mirror for\nthat reason alone, and then we can always experiment with a\n.gitlab-ci.yml in-repo.\n"},{"id":"360602","messageId":"nycvar.QRO.7.76.6.1810161111240.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"CAE5ih78gCgJgeXasdm6j+hO8Bk+_zk_-o5FqkMTr-q7fKDej9Q@mail.gmail.com","subject":"Re: [PATCH v2 09/13] git-p4: use `test_atexit` to kill the daemon","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-16T09:39:24Z","receivedAt":"2018-10-16T09:39:30Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Luke,\n\nOn Mon, 15 Oct 2018, Luke Diamand wrote:\n\n> On Mon, 15 Oct 2018 at 16:02, Johannes Schindelin\n> wrote:\n> >\n> > Hi Luke,\n> >\n> > On Mon, 15 Oct 2018, Luke Diamand wrote:\n> >\n> > > On Mon, 15 Oct 2018 at 11:12, Johannes Schindelin via GitGitGadget\n> > > wrote:\n> > > >\n> > > > From: Johannes Schindelin \n> > > >\n> > > > This should be more reliable than the current method, and prepares the\n> > > > test suite for a consistent way to clean up before re-running the tests\n> > > > with different options.\n> > > >\n> > >\n> > > I'm finding that it's leaving p4d processes lying around.\n> >\n> > That's a bummer!\n> >\n> > > e.g.\n> > >\n> > > $ ./t9820-git-p4-editor-handling.sh\n> > > \n> > > $ ./t9820-git-p4-editor-handling.sh\n> > > \n> >\n> > Since I do not currently have a setup with p4d installed, can you run that\n> > with `sh -x ...` and see whether this code path is hit?\n> \n> All you need to do is to put p4 and p4d in your PATH.\n> \n> https://www.perforce.com/downloads/helix-core-p4d\n> https://www.perforce.com/downloads/helix-command-line-client-p4\n\nI did download p4d.exe and p4.exe to $HOME/custom/p4/ (similar to the way\nci/install-dependencies.sh does it), from\n\n\thttp://filehost.perforce.com/perforce/r18.1/bin.ntx64/p4d.exe\n\thttp://filehost.perforce.com/perforce/r18.1/bin.ntx64/p4.exe\n\nand then prefixed PATH with $HOME/custom/p4, just to run t9820. However,\nit does not work here at all:\n\n-- snip --\n[...]\n++ git p4 clone '--dest=/usr/src/git/azure-pipelines/t/trash directory.t9820-git-p4-editor-handling/git' //depot\nUsage: git-p4 clone [options] //depot/path[@revRange]\n\n[...]\n-- snap --\n\nI *think* the reason for that is that the MSYS path mangling kicks in when\n`git.exe` is called with the `//depot` argument (the leading `//` can be\nused in MSYS and MSYS2 to indicate that the non-MSYS .exe wants an\nargument that starts with a single slash, but is not a path that needs to\nbe translated to a Windows path).\n\nSo I did the next best thing I could do: try things in the Windows\nSubsystem for Linux (AKA Bash on Ubuntu on Windows).\n\n> The server is free to use for a small number of users, you don't need\n> to do anything to make it go.\n> \n> \n> >\n> > test_done () {\n> > GIT_EXIT_OK=t\n> >\n> > + test -n \"$immediate\" || test_atexit_handler\n\nOn a slight tangent: I made up my mind on the `test -n \"$immediate\"` part:\nI will skip that. The daemon should be stopped also when `-i` was passed\nto the test script (to stop at the first failing test case). There is\nvery, very little use in keeping the daemon alive in that case.\n\nThe commit message of \"tests: introduce `test_atexit`\" already made the\ncase for that, but somehow I had not made up my mind yet.\n\n> > +\n> \n> + test -n\n> + test_atexit_handler\n> ./t9820-git-p4-editor-handling.sh: 764:\n> ./t9820-git-p4-editor-handling.sh: test_atexit_handler: not found\n> \n> Is that expected?\n\nNo, that is not expected. For me, it looks quite a bit different, though:\n\n-- snip --\n[...]\n+ test_done\n+ GIT_EXIT_OK=t\n+ test -n\n+ test_atexit_handler\n+ test : != { kill_p4d\n } && (exit \"$eval_ret\"); eval_ret=$?; :\n+ setup_malloc_check\n+ MALLOC_CHECK_=3 MALLOC_PERTURB_=165\n+ export MALLOC_CHECK_ MALLOC_PERTURB_\n+ test_eval_ { kill_p4d\n } && (exit \"$eval_ret\"); eval_ret=$?; :\n+ test_eval_inner_ { kill_p4d\n } && (exit \"$eval_ret\"); eval_ret=$?; :\n+ eval\n want_trace && set -x\n { kill_p4d\n } && (exit \"$eval_ret\"); eval_ret=$?; :\n+ want_trace\n+ test = t\n+ kill_p4d\n+ cat /home/wsl/git/wip/t/trash directory.t9820-git-p4-editor-handling/p4d.pid\n+ pid=20093\n+ retry_until_fail kill 20093\n+ time_in_seconds\n+ cd /\n+ /usr/bin/python -c import time; print(int(time.time()))\n+ timeout=1539681637\n+ kill 20093\n+ time_in_seconds\n+ cd /\n+ /usr/bin/python -c import time; print(int(time.time()))\n+ test 1539681577 -gt 1539681637\n+ sleep 1\n+ true\n+ time_in_seconds\n+ cd /\n+ /usr/bin/python -c import time; print(int(time.time()))\n+ test 1539681578 -gt 1539681871\n+ sleep 1\n+ kill 20093\n+ retry_until_fail kill -9 20093\n+ time_in_seconds\n+ cd /\n+ /usr/bin/python -c import time; print(int(time.time()))\n+ timeout=1539681638\n+ kill -9 20093\n+ test_must_fail kill 20093\n+ _test_ok=\n+ kill 20093\n+ exit_code=1\n+ test 1 -eq 0\n+ test_match_signal 13 1\n+ test 1 = 141\n+ test 1 = 269\n+ return 1\n+ test 1 -gt 129\n+ test 1 -eq 127\n+ test 1 -eq 126\n+ return 0\n[...]\n-- snap --\n\nAs you can see, it works quite well over here.\n\nMaybe I could trouble you to fetch the `vsts-ci` branch from\nhttps://github.com/dscho/git and test things on that one, just in case\nthat anything important got \"lost in the mail\"?\n\n> > if test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n> > then\n> >\n> > > And also\n> > >\n> > > $ ./t9800-git-p4-basic.sh\n> > > \n> > > Ctrl-C\n> >\n> > Oh, you're right.\n\nSo I tested this in WSL, and the relevant part of the output is this:\n\n-- snip --\n[...]\n+ eval test_prereq_lazily_UTF8_NFD_TO_NFC=$2\n+ test_prereq_lazily_UTF8_NFD_TO_NFC=\n # check whether FS converts nfd unicode to nfc\n au^CTraceback (most recent call last):\n File \"/home/virtualbox/git/wip/git-p4\", line 4143, in \n main()\n File \"/home/virtualbox/git/wip/git-p4\", line 4137, in main\n if not cmd.run(args):\n File \"/home/virtualbox/git/wip/git-p4\", line 3925, in run\n if not P4Sync.run(self, depotPaths):\n File \"/home/virtualbox/git/wip/git-p4\", line 3824, in run\n system([\"git\", \"symbolic-ref\", head_ref, self.branch])\n File \"/home/virtualbox/git/wip/git-p4\", line 278, in system\n retcode = subprocess.call(cmd, shell=expand)\n File \"/usr/lib/python2.7/subprocess.py\", line 523, in call\n return Popen(*popenargs, **kwargs).wait()\n File \"/usr/lib/python2.7/subprocess.py\", line 1392, in wait\n pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)\n File \"/usr/lib/python2.7/subprocess.py\", line 476, in _eintr_retry_call\n return func(*args)\nKeyboardInterrupt\n+ exit 130\n+ die\n+ code=130\n+ test_atexit_handler\n+ test : != { kill_p4d\n } && (exit \"$eval_ret\"); eval_ret=$?; :\n[...]\n-- snap --\n\nAs you can see, the Ctrl+C managed to call `die` very well, without my\nproposed change:\n\n> > I think I need to do something in this line:\n> >\n> > trap 'exit $?' INT\n> >\n> > in t/test-lib.sh, something like\n> >\n> > trap 'exit_code=$?; test_atexit_handler; exit $exit_code' INT\n> >\n> > would you agree? (And: could you test?)\n> \n> Not sure.\n\nI *think* what happens is that the INT trap already calls `exit 130`\nalright, and then the EXIT trap comes into effect, as planned.\n\nJust to make sure that it was not Python that caused the exit code 130, I\ntried it again, and luckily hit a spot where shell code was interpreted:\n\n-- snip --\n[...]\n+ test_prereq_lazily_LONG_IS_64BIT=\n test 8 -le \"$(build_option sizeof-long)\"\n\n+ test_lazy_^C+ exit 130\n+ die\n+ code=130\n+ test_atexit_handler\n+ test : != { kill_p4d\n } && (exit \"$eval_ret\"); eval_ret=$?; :\n[...]\n-- snap --\n\nSame here. The Ctrl+C triggers an `exit 130`, this time I am certain that\nit comes from the INT trap, and that `die` in turn is most definitely from\nthe EXIT trap, as intended.\n\nMy tests were performed with Dash, BTW, not with Bash:\n\n-- snip --\n$ dpkg -S $(which sh)\ndiversion by dash from: /bin/sh\ndiversion by dash to: /bin/sh.distrib\ndash: /bin/sh\n-- snap --\n\nBut I also repeated the tests with Bash, with almost the same result: the\nonly difference was that I did not manage to get the traces of\ntest_atexit_handler being called, but the p4d was no longer alive, so it\nmust have been killed as intended.\n\n> Send me a patch and I can try it out.\n\nI would *love* to figure out what is happening in your setup, and to fix\nit. In my hands, the patches work, though...\n\nCiao,\nDscho\n\n> \n> Thanks,\n> Luke\n> \n"},{"id":"360603","messageId":"20181016094016.GI19800@szeder.dev","threadId":"49270","inReplyTo":"0a53f371350146ecd01b44f0f42da447d7573689.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 13/13] travis: fix skipping tagged releases","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-16T09:40:16Z","receivedAt":"2018-10-16T09:40:22Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 15, 2018 at 03:12:17AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> From: Johannes Schindelin \n> \n> When building a PR, TRAVIS_BRANCH refers to the *target branch*.\n> Therefore, if a PR targets `master`, and `master` happened to be tagged,\n> we skipped the build by mistake.\n> \n> Fix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*)\n> when available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also\n> known as \"push builds\").\n\nThis all makes sense, but this patch is fixing a long-standing issue\nin our Travis CI build scripts (present since 09f5e9746c (travis-ci:\nskip a branch build if equal tag is present, 2017-09-10)), so it\nshould be the first in the series. So it could be picked up and\nperhaps even graduated faster than the rest of this patch series.\n\n> Signed-off-by: Johannes Schindelin \n> ---\n> ci/lib.sh | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 584abcd529..e1858ae609 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -3,7 +3,7 @@\n> if test true = \"$TRAVIS\"\n> then\n> \t# We are running within Travis CI\n> -\tCI_BRANCH=\"$TRAVIS_BRANCH\"\n> +\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> \tCI_COMMIT=\"$TRAVIS_COMMIT\"\n> \tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n> \tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n> -- \n> gitgitgadget\n"},{"id":"360604","messageId":"20181016094344.GJ19800@szeder.dev","threadId":"49270","inReplyTo":"486d1d25183fc7b0de599a0791ed41a87cc176cb.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 05/13] ci/lib.sh: add support for Azure Pipelines","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-16T09:43:44Z","receivedAt":"2018-10-16T09:43:51Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 15, 2018 at 03:12:05AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> From: Johannes Schindelin \n> \n> This patch introduces a conditional arm that defines some environment\n> variables and a function that displays the URL given the job id (to\n> identify previous runs for known-good trees).\n> \n> Signed-off-by: Johannes Schindelin \n> ---\n> ci/lib.sh | 23 +++++++++++++++++++++++\n> 1 file changed, 23 insertions(+)\n> \n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 8532555b4e..584abcd529 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -19,6 +19,29 @@ then\n> \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n\nSo we install these two additional packages in the macOS build jobs on\nTravis CI ...\n\n> \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n> \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n> +elif test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n> +then\n> +\t# We are running in Azure Pipelines\n> +\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n> +\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n> +\tCI_JOB_ID=\"$BUILD_BUILDID\"\n> +\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n> +\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n> +\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n> +\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n> +\tCC=\"${CC:-gcc}\"\n> +\n> +\t# use a subdirectory of the cache dir (because the file share is shared\n> +\t# among *all* phases)\n> +\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n> +\n> +\turl_for_job_id () {\n> +\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n> +\t}\n> +\n> +\tBREW_INSTALL_PACKAGES=\n\n... but not on Azure Pipelines. Is this mere oversight or\nintentional? If it's intentional, then I think the commit message\nshould mention why.\n\n> +\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> +\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n> fi\n> \n> skip_branch_tip_with_tag () {\n> -- \n> gitgitgadget\n> \n"},{"id":"360605","messageId":"nycvar.QRO.7.76.6.1810161155030.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqsh16zc5x.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 06/13] Add a build definition for Azure DevOps","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-16T09:58:03Z","receivedAt":"2018-10-16T09:58:07Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 16 Oct 2018, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > Also, we make use of the shiny new feature we just introduced where the\n> > test suite can output JUnit-style .xml files. This information is made\n> > available in a nice UI that allows the viewer to filter by phase and/or\n> > test number, and to see trends such as: number of (failing) tests, time\n> > spent running the test suite, etc.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > azure-pipelines.yml | 319 ++++++++++++++++++++++++++++++++++++++++++\n> > ci/mount-fileshare.sh | 26 ++++\n> > 2 files changed, 345 insertions(+)\n> > create mode 100644 azure-pipelines.yml\n> > create mode 100755 ci/mount-fileshare.sh\n> \n> I wonder if there is a need to keep what is tested by this and\n> Travis in sync in any way, but most of the logic is not defined in\n> these \"steps\" but implemented in ci/*.sh scripts to be shared, so it\n> would be OK, I guess.\n\nIndeed, that was my intention.\n\nThese ci scripts are not only useful for Travis and Azure Pipelines, after\nall, but also a good documentation how to test locally. For example, to\nrepeat Luke's Perforce testing, I could simply use the URLs listed in\nthose ci scripts to get almost the same setup locally (close enough for\ntesting).\n\nSo it is not only about sharing, and ease of maintenance, but it is also\nabout documenting.\n\nAnd yes, sharing means that we do not have to waste brain cycles on\nkeeping things in sync.\n\n> > diff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\n> > new file mode 100755\n> > index 0000000000..5fb5f74b70\n> > --- /dev/null\n> > +++ b/ci/mount-fileshare.sh\n> > @@ -0,0 +1,26 @@\n> > +#!/bin/sh\n> > +\n> > +die () {\n> > +\techo \"$*\" >&2\n> > +\texit 1\n> > +}\n> > +\n> > +test $# = 4 ||\n> > +die \"Usage: $0 \n> Missing closing '>'.\n\nThanks!\n\n> > +\n> > +mkdir -p \"$4\" || die \"Could not create $4\"\n> > +\n> > +case \"$(uname -s)\" in\n> > +Linux)\n> > +\tsudo mount -t cifs -o vers=3.0,username=\"$2\",password=\"$3\",dir_mode=0777,file_mode=0777,serverino \"$1\" \"$4\"\n> > +\t;;\n> > +Darwin)\n> > +\tpass=\"$(echo \"$3\" | sed -e 's/\\//%2F/g' -e 's/+/%2B/g')\" &&\n> > +\tmount -t smbfs,soft \"smb://$2:$pass@${1#//}\" \"$4\"\n> > +\t;;\n> > +*)\n> > +\tdie \"No support for $(uname -s)\"\n> > +\t;;\n> > +esac ||\n> > +die \"Could not mount $4\"\n> > +\n> \n> Trailing blank line.\n> \n> Thanks.\n\nThank you, both issues will be fixed in the next iteration,\nDscho\n"},{"id":"360606","messageId":"20181016100438.GK19800@szeder.dev","threadId":"49270","inReplyTo":"ae3c42519abff7ef32c767f9587ef7f0160033ed.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 10/13] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-16T10:04:38Z","receivedAt":"2018-10-16T10:04:44Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 15, 2018 at 03:12:12AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> From: Johannes Schindelin \n> \n> The JUnit XML format lends itself to be presented in a powerful UI,\n> where you can drill down to the information you are interested in very\n> quickly.\n> \n> For test failures, this usually means that you want to see the detailed\n> trace of the failing tests.\n> \n> With Travis CI, we passed the `--verbose-log` option to get those\n> traces. However, that seems excessive, as we do not need/use the logs in\n\nAs someone who has dug into a few occasional failures found by Travis\nCI, I'd say that the output of '--verbose-log -x' is not excessive,\nbut downright essential.\n\n> almost all of those cases: only when a test fails do we have a way to\n> include the trace.\n> \n> So let's do something different when using Azure DevOps: let's run all\n> the tests with `--quiet` first, and only if a failure is encountered,\n> try to trace the commands as they are executed.\n> \n> Of course, we cannot turn on `--verbose-log` after the fact. So let's\n> just re-run the test with all the same options, adding `--verbose-log`.\n> And then munging the output file into the JUnit XML on the fly.\n> \n> Note: there is an off chance that re-running the test in verbose mode\n> \"fixes\" the failures (and this does happen from time to time!). That is\n> a possibility we should be able to live with.\n\nAny CI system worth its salt should provide as much information about\nany failures as possible, especially when it was lucky enough to\nstumble upon a rare and hard to reproduce non-deterministic failure.\n\n> Ideally, we would label\n> this as \"Passed upon rerun\", and Azure Pipelines even know about that\n> outcome, but it is not available when using the JUnit XML format for\n> now:\n> https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n> \n> Signed-off-by: Johannes Schindelin \n"},{"id":"360608","messageId":"20181016103029.GL19800@szeder.dev","threadId":"49270","inReplyTo":"20181015142215.GB1764@syl.hsd1.wa.comcast.net","subject":"Re: [PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-16T10:30:29Z","receivedAt":"2018-10-16T10:30:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 15, 2018 at 07:22:15AM -0700, Taylor Blau wrote:\n> Would we like to abandon Travis as our main CI service for upstream\n> git.git, and build on Azure Pipelines only?\n\nIt's not only about \"upstream git.git\", but also about contributors,\nwho might have enabled Travis CI integration on their forks on GitHub.\nHaving a '.travis.yml' and associated 'ci/*' scripts in git.git makes\nit possible for them to easily build and test their branches on their\nown.\n\n"},{"id":"360610","messageId":"CACBZZX42Y2LAUsNT0dcmDy7ws+MaXtShD8K1YoUXx2kEBLH+aQ@mail.gmail.com","threadId":"49270","inReplyTo":"20181016025328.GC90754@syl","subject":"Re: [PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-16T11:55:06Z","receivedAt":"2018-10-16T11:55:20Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Tue, Oct 16, 2018 at 4:55 AM Taylor Blau wrote:\n>\n> On Mon, Oct 15, 2018 at 04:55:25PM +0200, Johannes Schindelin wrote:\n> > Another really good reason for me to do this is that I can prod the Azure\n> > Pipelines team directly. And I even get an answer, usually within minutes.\n> > Which is a lot faster than the Travis team answers my questions, which\n> > is... not yet? (I tried to get in contact with them in late 2015 or early\n> > 2016, and I tried again a year later, and then a couple of months later,\n> > and I have yet to hear back.)\n>\n> Certainly a good reason. To be clear/fair, I've sent in a number of\n> support tickets to Travis CI over the years, and have always been\n> responded to in a short amount of time with helpful answers. I think\n> that we would really be fine in either case, TBH.\n\nWas this in the context of \"I'm this random dude using Travis\" or \"as\nyou guys know I work for GitHub, your biggest? customer...\" ? :)\n"},{"id":"360617","messageId":"nycvar.QRO.7.76.6.1810161445300.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"20181016094016.GI19800@szeder.dev","subject":"Re: [PATCH v2 13/13] travis: fix skipping tagged releases","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-16T12:45:52Z","receivedAt":"2018-10-16T12:45:56Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 16 Oct 2018, SZEDER Gábor wrote:\n\n> On Mon, Oct 15, 2018 at 03:12:17AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> > From: Johannes Schindelin \n> > \n> > When building a PR, TRAVIS_BRANCH refers to the *target branch*.\n> > Therefore, if a PR targets `master`, and `master` happened to be tagged,\n> > we skipped the build by mistake.\n> > \n> > Fix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*)\n> > when available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also\n> > known as \"push builds\").\n> \n> This all makes sense, but this patch is fixing a long-standing issue\n> in our Travis CI build scripts (present since 09f5e9746c (travis-ci:\n> skip a branch build if equal tag is present, 2017-09-10)), so it\n> should be the first in the series. So it could be picked up and\n> perhaps even graduated faster than the rest of this patch series.\n\nMakes sense.\n\nThanks,\nDscho\n\n> \n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > ci/lib.sh | 2 +-\n> > 1 file changed, 1 insertion(+), 1 deletion(-)\n> > \n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index 584abcd529..e1858ae609 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -3,7 +3,7 @@\n> > if test true = \"$TRAVIS\"\n> > then\n> > \t# We are running within Travis CI\n> > -\tCI_BRANCH=\"$TRAVIS_BRANCH\"\n> > +\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> > \tCI_COMMIT=\"$TRAVIS_COMMIT\"\n> > \tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n> > \tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n> > -- \n> > gitgitgadget\n> "},{"id":"360618","messageId":"nycvar.QRO.7.76.6.1810161452040.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"20181016094344.GJ19800@szeder.dev","subject":"Re: [PATCH v2 05/13] ci/lib.sh: add support for Azure Pipelines","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-16T12:53:21Z","receivedAt":"2018-10-16T12:53:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Gábor,\n\nOn Tue, 16 Oct 2018, SZEDER Gábor wrote:\n\n> On Mon, Oct 15, 2018 at 03:12:05AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> > From: Johannes Schindelin \n> > \n> > This patch introduces a conditional arm that defines some environment\n> > variables and a function that displays the URL given the job id (to\n> > identify previous runs for known-good trees).\n> > \n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > ci/lib.sh | 23 +++++++++++++++++++++++\n> > 1 file changed, 23 insertions(+)\n> > \n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index 8532555b4e..584abcd529 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -19,6 +19,29 @@ then\n> > \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n> \n> So we install these two additional packages in the macOS build jobs on\n> Travis CI ...\n> \n> > \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n> > \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n> > +elif test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n> > +then\n> > +\t# We are running in Azure Pipelines\n> > +\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n> > +\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n> > +\tCI_JOB_ID=\"$BUILD_BUILDID\"\n> > +\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n> > +\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n> > +\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n> > +\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n> > +\tCC=\"${CC:-gcc}\"\n> > +\n> > +\t# use a subdirectory of the cache dir (because the file share is shared\n> > +\t# among *all* phases)\n> > +\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n> > +\n> > +\turl_for_job_id () {\n> > +\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n> > +\t}\n> > +\n> > +\tBREW_INSTALL_PACKAGES=\n> \n> ... but not on Azure Pipelines. Is this mere oversight or\n> intentional? If it's intentional, then I think the commit message\n> should mention why.\n\nBoth packages are already available on Azure Pipelines' VMs. I will\nmention this in the commit message.\n\nThanks,\nDscho\n\n> \n> > +\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> > +\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n> > fi\n> > \n> > skip_branch_tip_with_tag () {\n> > -- \n> > gitgitgadget\n> > \n> "},{"id":"360620","messageId":"nycvar.QRO.7.76.6.1810161455200.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"20181016100438.GK19800@szeder.dev","subject":"Re: [PATCH v2 10/13] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-16T13:02:38Z","receivedAt":"2018-10-16T13:02:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Gábor,\n\nOn Tue, 16 Oct 2018, SZEDER Gábor wrote:\n\n> On Mon, Oct 15, 2018 at 03:12:12AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> > From: Johannes Schindelin \n> > \n> > The JUnit XML format lends itself to be presented in a powerful UI,\n> > where you can drill down to the information you are interested in very\n> > quickly.\n> > \n> > For test failures, this usually means that you want to see the detailed\n> > trace of the failing tests.\n> > \n> > With Travis CI, we passed the `--verbose-log` option to get those\n> > traces. However, that seems excessive, as we do not need/use the logs in\n> \n> As someone who has dug into a few occasional failures found by Travis\n> CI, I'd say that the output of '--verbose-log -x' is not excessive,\n> but downright essential.\n\nI agree that the output is essential for drilling down into failures. This\nparagraph, however, talks about the general case: where there are *no*\nfailures. See here:\n\n> > almost all of those cases: only when a test fails do we have a way to\n> > include the trace.\n> > \n> > So let's do something different when using Azure DevOps: let's run all\n> > the tests with `--quiet` first, and only if a failure is encountered,\n> > try to trace the commands as they are executed.\n> > \n> > Of course, we cannot turn on `--verbose-log` after the fact. So let's\n> > just re-run the test with all the same options, adding `--verbose-log`.\n> > And then munging the output file into the JUnit XML on the fly.\n> > \n> > Note: there is an off chance that re-running the test in verbose mode\n> > \"fixes\" the failures (and this does happen from time to time!). That is\n> > a possibility we should be able to live with.\n> \n> Any CI system worth its salt should provide as much information about\n> any failures as possible, especially when it was lucky enough to\n> stumble upon a rare and hard to reproduce non-deterministic failure.\n\nI would agree with you if more people started to pay attention to our CI\nfailures. And if we had some sort of a development model where a CI\nfailure would halt development on that particular topic until the failure\nis fixed, with the responsibility assigned to somebody to fix it.\n\nThis is not the case here, though. pu is broken for ages, at least on\nWindows, and even a *single* topic is enough to do that. And this is even\nworse with flakey tests. I cannot remember *how often* I saw CI failures\nin t5570-git-daemon.sh, for example. It is rare enough that it is obvious\nthat this is a problem of the *regression test*, rather than a problem of\nthe code that is to be tested.\n\nSo I would suggest to go forward with my proposed strategy for the moment,\nright up until the time when we have had the resources to fix t5570, for\nstarters.\n\nCiao,\nDscho\n\n> > Ideally, we would label this as \"Passed upon rerun\", and Azure\n> > Pipelines even know about that outcome, but it is not available when\n> > using the JUnit XML format for now:\n> > https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n> > \n> > Signed-off-by: Johannes Schindelin \n> "},{"id":"360640","messageId":"20181016160348.GM19800@szeder.dev","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1810161455200.4546@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 10/13] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-16T16:03:48Z","receivedAt":"2018-10-16T16:03:56Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Oct 16, 2018 at 03:02:38PM +0200, Johannes Schindelin wrote:\n> Hi Gábor,\n> \n> On Tue, 16 Oct 2018, SZEDER Gábor wrote:\n> \n> > On Mon, Oct 15, 2018 at 03:12:12AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> > > From: Johannes Schindelin \n> > > \n> > > The JUnit XML format lends itself to be presented in a powerful UI,\n> > > where you can drill down to the information you are interested in very\n> > > quickly.\n> > > \n> > > For test failures, this usually means that you want to see the detailed\n> > > trace of the failing tests.\n> > > \n> > > With Travis CI, we passed the `--verbose-log` option to get those\n> > > traces. However, that seems excessive, as we do not need/use the logs in\n> > \n> > As someone who has dug into a few occasional failures found by Travis\n> > CI, I'd say that the output of '--verbose-log -x' is not excessive,\n> > but downright essential.\n> \n> I agree that the output is essential for drilling down into failures. This\n> paragraph, however, talks about the general case: where there are *no*\n> failures. See here:\n\nBut you don't know in advance whether there will be any failures or\nnot, so it only makes sense to run all tests with '--verbose-log -x'\nby default, just in case a Heisenbug decides to make an appearance.\n\n> > > almost all of those cases: only when a test fails do we have a way to\n> > > include the trace.\n> > > \n> > > So let's do something different when using Azure DevOps: let's run all\n> > > the tests with `--quiet` first, and only if a failure is encountered,\n> > > try to trace the commands as they are executed.\n> > > \n> > > Of course, we cannot turn on `--verbose-log` after the fact. So let's\n> > > just re-run the test with all the same options, adding `--verbose-log`.\n> > > And then munging the output file into the JUnit XML on the fly.\n> > > \n> > > Note: there is an off chance that re-running the test in verbose mode\n> > > \"fixes\" the failures (and this does happen from time to time!). That is\n> > > a possibility we should be able to live with.\n> > \n> > Any CI system worth its salt should provide as much information about\n> > any failures as possible, especially when it was lucky enough to\n> > stumble upon a rare and hard to reproduce non-deterministic failure.\n> \n> I would agree with you if more people started to pay attention to our CI\n> failures. And if we had some sort of a development model where a CI\n> failure would halt development on that particular topic until the failure\n> is fixed, with the responsibility assigned to somebody to fix it.\n> \n> This is not the case here, though. pu is broken for ages, at least on\n> Windows, and even a *single* topic is enough to do that. And this is even\n> worse with flakey tests. I cannot remember *how often* I saw CI failures\n> in t5570-git-daemon.sh, for example. It is rare enough that it is obvious\n> that this is a problem of the *regression test*, rather than a problem of\n> the code that is to be tested.\n\nSome occasional failures in t5570 are actually caused by issues in Git\non certain platforms:\n\n https://public-inbox.org/git/CAM0VKj=MCS+cmOgzf_XyPeb+qZrFmuMH52-PV_NDMZA9X+rRoA@mail.gmail.com/T/#u\n\n> So I would suggest to go forward with my proposed strategy for the moment,\n> right up until the time when we have had the resources to fix t5570, for\n> starters.\n\nI don't really understand what the occasional failures in t5570 have\nto do with the amount of information a CI system should gather about\nfailures in general. Or how many people pay attention to it, or what\nkind of development model we have, for that matter. The way I see it\nthese are unrelated issues, and a CI system should always provide as\nmuch information about failures as possible. If only a few people pay\nattention to it, then for the sake of those few.\n\n\n> > > Ideally, we would label this as \"Passed upon rerun\", and Azure\n> > > Pipelines even know about that outcome, but it is not available when\n> > > using the JUnit XML format for now:\n> > > https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n> > > \n> > > Signed-off-by: Johannes Schindelin \n> > \n\n"},{"id":"360658","messageId":"20181016191250.GO19800@szeder.dev","threadId":"49270","inReplyTo":"1a22efe849d6da79f2c639c62a1483361a130238.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/13] Add a build definition for Azure DevOps","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-16T19:12:50Z","receivedAt":"2018-10-16T19:12:57Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 15, 2018 at 03:12:06AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> diff --git a/azure-pipelines.yml b/azure-pipelines.yml\n> new file mode 100644\n> index 0000000000..b5749121d2\n> --- /dev/null\n> +++ b/azure-pipelines.yml\n> @@ -0,0 +1,319 @@\n> +resources:\n> +- repo: self\n> + fetchDepth: 1\n> +\n> +phases:\n> +- phase: linux_clang\n> + displayName: linux-clang\n> + condition: succeeded()\n> + queue:\n> + name: Hosted Ubuntu 1604\n> + steps:\n> + - bash: |\n> + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n> +\n> + sudo apt-get update &&\n> + sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin &&\n> +\n> + export CC=clang || exit 1\n> +\n> + ci/install-dependencies.sh\n\nI think you would want to 'exit 1' when this script fails.\nThis applies to other build jobs (erm, phases?) below as well.\n\n> + ci/run-build-and-tests.sh || {\n> + ci/print-test-failures.sh\n> + exit 1\n> + }\n> +\n> + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n> + displayName: 'ci/run-build-and-tests.sh'\n> + env:\n> + GITFILESHAREPWD: $(gitfileshare.pwd)\n> + - task: PublishTestResults@2\n> + displayName: 'Publish Test Results **/TEST-*.xml'\n> + inputs:\n> + mergeTestResults: true\n> + testRunTitle: 'linux-clang'\n> + platform: Linux\n> + publishRunAttachments: false\n> + condition: succeededOrFailed()\n> +\n> +- phase: linux_gcc\n> + displayName: linux-gcc\n> + condition: succeeded()\n> + queue:\n> + name: Hosted Ubuntu 1604\n> + steps:\n> + - bash: |\n> + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n> +\n> + sudo apt-get update &&\n> + sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin || exit 1\n> +\n\nOn Travis CI the Linux GCC build job uses gcc-8 instead of whatever\nthe default is in that old-ish Ubuntu LTS; see 37fa4b3c78 (travis-ci:\nrun gcc-8 on linux-gcc jobs, 2018-05-19).\n\n> + ci/install-dependencies.sh\n> + ci/run-build-and-tests.sh || {\n> + ci/print-test-failures.sh\n> + exit 1\n> + }\n> +\n> + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n> + displayName: 'ci/run-build-and-tests.sh'\n> + env:\n> + GITFILESHAREPWD: $(gitfileshare.pwd)\n> + - task: PublishTestResults@2\n> + displayName: 'Publish Test Results **/TEST-*.xml'\n> + inputs:\n> + mergeTestResults: true\n> + testRunTitle: 'linux-gcc'\n> + platform: Linux\n> + publishRunAttachments: false\n> + condition: succeededOrFailed()\n> +\n> +- phase: osx_clang\n> + displayName: osx-clang\n> + condition: succeeded()\n> + queue:\n> + name: Hosted macOS\n> + steps:\n> + - bash: |\n> + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n> +\n> + export CC=clang\n> +\n> + ci/install-dependencies.sh\n> + ci/run-build-and-tests.sh || {\n> + ci/print-test-failures.sh\n> + exit 1\n> + }\n> +\n> + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n> + displayName: 'ci/run-build-and-tests.sh'\n> + env:\n> + GITFILESHAREPWD: $(gitfileshare.pwd)\n> + - task: PublishTestResults@2\n> + displayName: 'Publish Test Results **/TEST-*.xml'\n> + inputs:\n> + mergeTestResults: true\n> + testRunTitle: 'osx-clang'\n> + platform: macOS\n> + publishRunAttachments: false\n> + condition: succeededOrFailed()\n> +\n> +- phase: osx_gcc\n> + displayName: osx-gcc\n> + condition: succeeded()\n> + queue:\n> + name: Hosted macOS\n> + steps:\n> + - bash: |\n> + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n> +\n\nHere you should 'export CC=gcc', because on macOS 'cc' is 'clang' by\ndefault.\n\nNote, however, that setting 'CC' in the environment alone has no\neffect on the build process, it will still use 'cc'. Keep an eye on\nwhere this thread will lead to:\n\n https://public-inbox.org/git/20181016184537.GN19800@szeder.dev/T/#u\n\n> + ci/install-dependencies.sh\n> + ci/run-build-and-tests.sh || {\n> + ci/print-test-failures.sh\n> + exit 1\n> + }\n> +\n> + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n> + displayName: 'ci/run-build-and-tests.sh'\n> + env:\n> + GITFILESHAREPWD: $(gitfileshare.pwd)\n> + - task: PublishTestResults@2\n> + displayName: 'Publish Test Results **/TEST-*.xml'\n> + inputs:\n> + mergeTestResults: true\n> + testRunTitle: 'osx-gcc'\n> + platform: macOS\n> + publishRunAttachments: false\n> + condition: succeededOrFailed()\n> +\n"},{"id":"360665","messageId":"nycvar.QRO.7.76.6.1810162251380.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"20181016160348.GM19800@szeder.dev","subject":"Re: [PATCH v2 10/13] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-16T20:53:31Z","receivedAt":"2018-10-16T20:53:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Gábor,\n\nOn Tue, 16 Oct 2018, SZEDER Gábor wrote:\n\n> On Tue, Oct 16, 2018 at 03:02:38PM +0200, Johannes Schindelin wrote:\n>\n> > So I would suggest to go forward with my proposed strategy for the\n> > moment, right up until the time when we have had the resources to fix\n> > t5570, for starters.\n> \n> I don't really understand what the occasional failures in t5570 have\n> to do with the amount of information a CI system should gather about\n> failures in general.\n\nI see it plenty of times that too many CI failures essentially render\nevery developer numb.\n\nIf every 3rd CI run causes a failure, and seemingly every of these\nfailures indicates a mistake in the regression test, rather than a\nregression, developers stop paying attention.\n\nWhich is the exact opposite of what I want to achieve here.\n\nCiao,\nDscho"},{"id":"360756","messageId":"nycvar.QRO.7.76.6.1810171512100.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"20181016191250.GO19800@szeder.dev","subject":"Re: [PATCH v2 06/13] Add a build definition for Azure DevOps","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-17T14:58:21Z","receivedAt":"2018-10-17T14:58:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Gábor,\n\nOn Tue, 16 Oct 2018, SZEDER Gábor wrote:\n\n> On Mon, Oct 15, 2018 at 03:12:06AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> > diff --git a/azure-pipelines.yml b/azure-pipelines.yml\n> > new file mode 100644\n> > index 0000000000..b5749121d2\n> > --- /dev/null\n> > +++ b/azure-pipelines.yml\n> > @@ -0,0 +1,319 @@\n> > +resources:\n> > +- repo: self\n> > + fetchDepth: 1\n> > +\n> > +phases:\n> > +- phase: linux_clang\n> > + displayName: linux-clang\n> > + condition: succeeded()\n> > + queue:\n> > + name: Hosted Ubuntu 1604\n> > + steps:\n> > + - bash: |\n> > + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n> > +\n> > + sudo apt-get update &&\n> > + sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin &&\n> > +\n> > + export CC=clang || exit 1\n> > +\n> > + ci/install-dependencies.sh\n> \n> I think you would want to 'exit 1' when this script fails.\n> This applies to other build jobs (erm, phases?) below as well.\n\nTrue.\n\nFWIW the nomenclature is \"build\" or \"job\" or \"build job\" for the entire\nrun, from what I understand. The \"phase\" is the individual chunk that is\nrun in an individual agent, i.e. you can have a single job running test on\ndifferent OSes in separate phases.\n\n> > + ci/run-build-and-tests.sh || {\n> > + ci/print-test-failures.sh\n> > + exit 1\n> > + }\n> > +\n> > + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n> > + displayName: 'ci/run-build-and-tests.sh'\n> > + env:\n> > + GITFILESHAREPWD: $(gitfileshare.pwd)\n> > + - task: PublishTestResults@2\n> > + displayName: 'Publish Test Results **/TEST-*.xml'\n> > + inputs:\n> > + mergeTestResults: true\n> > + testRunTitle: 'linux-clang'\n> > + platform: Linux\n> > + publishRunAttachments: false\n> > + condition: succeededOrFailed()\n> > +\n> > +- phase: linux_gcc\n> > + displayName: linux-gcc\n> > + condition: succeeded()\n> > + queue:\n> > + name: Hosted Ubuntu 1604\n> > + steps:\n> > + - bash: |\n> > + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n> > +\n> > + sudo apt-get update &&\n> > + sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin || exit 1\n> > +\n> \n> On Travis CI the Linux GCC build job uses gcc-8 instead of whatever\n> the default is in that old-ish Ubuntu LTS; see 37fa4b3c78 (travis-ci:\n> run gcc-8 on linux-gcc jobs, 2018-05-19).\n\nI'll add those dependencies explicitly. It does seem, however, from a\ncursory look at the log, that gcc-8 should not even be picked up, as it is\nset via the environment variable `CC` (which, as you point out in\nbelow-referenced thread, is not respected):\n\n[...]\n2018-10-16T10:00:36.0177072Z ++ '[' linux-gcc = linux-gcc ']'\n2018-10-16T10:00:36.0177380Z ++ export CC=gcc-8\n2018-10-16T10:00:36.0177630Z ++ CC=gcc-8\n2018-10-16T10:00:36.0177917Z ++ case \"$jobname\"\n[...]\n\n(see https://dev.azure.com/git/git/_build/results?buildId=192&view=logs)\n\n> > + ci/install-dependencies.sh\n> > + ci/run-build-and-tests.sh || {\n> > + ci/print-test-failures.sh\n> > + exit 1\n> > + }\n> > +\n> > + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n> > + displayName: 'ci/run-build-and-tests.sh'\n> > + env:\n> > + GITFILESHAREPWD: $(gitfileshare.pwd)\n> > + - task: PublishTestResults@2\n> > + displayName: 'Publish Test Results **/TEST-*.xml'\n> > + inputs:\n> > + mergeTestResults: true\n> > + testRunTitle: 'linux-gcc'\n> > + platform: Linux\n> > + publishRunAttachments: false\n> > + condition: succeededOrFailed()\n> > +\n> > +- phase: osx_clang\n> > + displayName: osx-clang\n> > + condition: succeeded()\n> > + queue:\n> > + name: Hosted macOS\n> > + steps:\n> > + - bash: |\n> > + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n> > +\n> > + export CC=clang\n> > +\n> > + ci/install-dependencies.sh\n> > + ci/run-build-and-tests.sh || {\n> > + ci/print-test-failures.sh\n> > + exit 1\n> > + }\n> > +\n> > + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n> > + displayName: 'ci/run-build-and-tests.sh'\n> > + env:\n> > + GITFILESHAREPWD: $(gitfileshare.pwd)\n> > + - task: PublishTestResults@2\n> > + displayName: 'Publish Test Results **/TEST-*.xml'\n> > + inputs:\n> > + mergeTestResults: true\n> > + testRunTitle: 'osx-clang'\n> > + platform: macOS\n> > + publishRunAttachments: false\n> > + condition: succeededOrFailed()\n> > +\n> > +- phase: osx_gcc\n> > + displayName: osx-gcc\n> > + condition: succeeded()\n> > + queue:\n> > + name: Hosted macOS\n> > + steps:\n> > + - bash: |\n> > + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n> > +\n> \n> Here you should 'export CC=gcc', because on macOS 'cc' is 'clang' by\n> default.\n> \n> Note, however, that setting 'CC' in the environment alone has no\n> effect on the build process, it will still use 'cc'. Keep an eye on\n> where this thread will lead to:\n> \n> https://public-inbox.org/git/20181016184537.GN19800@szeder.dev/T/#u\n\nWill do.\n\nThanks,\nDscho\n\n> \n> > + ci/install-dependencies.sh\n> > + ci/run-build-and-tests.sh || {\n> > + ci/print-test-failures.sh\n> > + exit 1\n> > + }\n> > +\n> > + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n> > + displayName: 'ci/run-build-and-tests.sh'\n> > + env:\n> > + GITFILESHAREPWD: $(gitfileshare.pwd)\n> > + - task: PublishTestResults@2\n> > + displayName: 'Publish Test Results **/TEST-*.xml'\n> > + inputs:\n> > + mergeTestResults: true\n> > + testRunTitle: 'osx-gcc'\n> > + platform: macOS\n> > + publishRunAttachments: false\n> > + condition: succeededOrFailed()\n> > +\n> "},{"id":"360794","messageId":"20181017232952.GT19800@szeder.dev","threadId":"49270","inReplyTo":"3e2193a73de0b68d5a38f1792642c828f4aed1db.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/13] git-p4: use `test_atexit` to kill the daemon","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-17T23:29:52Z","receivedAt":"2018-10-17T23:29:59Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 15, 2018 at 03:12:11AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> From: Johannes Schindelin \n> \n> This should be more reliable than the current method, and prepares the\n> test suite for a consistent way to clean up before re-running the tests\n> with different options.\n\nThis patch makes the test suite hang in all four Travis CI build jobs\nwith P4 installed without any of the P4 tests finishing.\n\nReverting this patch from the whole patch series makes it work again.\n\nI've also tried to revert only this first hunk of the patch below,\nbecause based on the comment I thought it's worth a try, but it didn't\nreally help. It did make a difference: the 300s watchdog timer\neventually kicked in, and then the test scripts could finish\nsuccessfully... but there are a lot of P4 test scripts, and with each\ntaking 300s the build job still timeouted.\n\nAll this may (or may not) be related to and be a different symptom of\nthe leftover p4d processes Luke mentioned. I couldn't reproduce any\nof this on my machine so far.\n\n> diff --git a/t/lib-git-p4.sh b/t/lib-git-p4.sh\n> index c27599474c..f4f5d7d296 100644\n> --- a/t/lib-git-p4.sh\n> +++ b/t/lib-git-p4.sh\n> @@ -74,15 +74,6 @@ cli=\"$TRASH_DIRECTORY/cli\"\n> git=\"$TRASH_DIRECTORY/git\"\n> pidfile=\"$TRASH_DIRECTORY/p4d.pid\"\n> \n> -# Sometimes \"prove\" seems to hang on exit because p4d is still running\n> -cleanup () {\n> -\tif test -f \"$pidfile\"\n> -\tthen\n> -\t\tkill -9 $(cat \"$pidfile\") 2>/dev/null && exit 255\n> -\tfi\n> -}\n> -trap cleanup EXIT\n> -\n> # git p4 submit generates a temp file, which will\n> # not get cleaned up if the submission fails. Don't\n> # clutter up /tmp on the test machine.\n"},{"id":"360804","messageId":"xmqqo9bsvwn9.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"CACBZZX59vT1CTiUPfPZDJUSRH=NSzmWiabux=KRdTUFa1R=5bw@mail.gmail.com","subject":"Re: [PATCH 0/9] Offer to run CI/PR builds in Visual Studio Team Services","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-18T02:03:38Z","receivedAt":"2018-10-18T02:03:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n>> sites could do the same polling and mirroring. I am just too lazy\n>> to open a new account at yet another hosting site to add that for\n>> loop, but I may choose to when I am absolutely bored and nothing\n>> else to do ;-).\n>\n> Do you mind if I squat gitlab.com/git/git in the meantime (i.e. create\n> an org etc.) and have it mirror github.com/git/git?, I'll hand the\n\nObviously somebody who is not even interested in obtaining an\naccount would appreciate, not just \"would not mind\", if a trusted\nmember in the community did that for the community ;-)\n\nThanks.\n"},{"id":"360880","messageId":"20181018220106.GU19800@szeder.dev","threadId":"49270","inReplyTo":"815152e0f57d545ae2fae7429c16e5be497746be.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 02/13] ci/lib.sh: encapsulate Travis-specific things","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-18T22:01:06Z","receivedAt":"2018-10-18T22:01:13Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 15, 2018 at 03:12:00AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 06970f7213..8532555b4e 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -1,5 +1,26 @@\n> # Library of functions shared by all CI scripts\n> \n> +if test true = \"$TRAVIS\"\n> +then\n> +\t# We are running within Travis CI\n> +\tCI_BRANCH=\"$TRAVIS_BRANCH\"\n> +\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n> +\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n> +\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n> +\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n> +\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n> +\n> +\tcache_dir=\"$HOME/travis-cache\"\n> +\n> +\turl_for_job_id () {\n> +\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n> +\t}\n> +\n> +\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n> +\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n> +\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n> +fi\n\nPlease set all these variables ...\n\n> +\n> skip_branch_tip_with_tag () {\n> \t# Sometimes, a branch is pushed at the same time the tag that points\n> \t# at the same commit as the tip of the branch is pushed, and building\n> @@ -13,10 +34,10 @@ skip_branch_tip_with_tag () {\n> \t# we can skip the build because we won't be skipping a build\n> \t# of a tag.\n> \n> -\tif TAG=$(git describe --exact-match \"$TRAVIS_BRANCH\" 2>/dev/null) &&\n> -\t\ttest \"$TAG\" != \"$TRAVIS_BRANCH\"\n> +\tif TAG=$(git describe --exact-match \"$CI_BRANCH\" 2>/dev/null) &&\n> +\t\ttest \"$TAG\" != \"$CI_BRANCH\"\n> \tthen\n> -\t\techo \"$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)\"\n> +\t\techo \"$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)\"\n> \t\texit 0\n> \tfi\n> }\n> @@ -25,7 +46,7 @@ skip_branch_tip_with_tag () {\n> # job if we encounter the same tree again and can provide a useful info\n> # message.\n> save_good_tree () {\n> -\techo \"$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID\" >>\"$good_trees_file\"\n> +\techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n> \t# limit the file size\n> \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n> \tmv \"$good_trees_file\".tmp \"$good_trees_file\"\n> @@ -35,7 +56,7 @@ save_good_tree () {\n> # successfully before (e.g. because the branch got rebased, changing only\n> # the commit messages).\n> skip_good_tree () {\n> -\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $TRAVIS_COMMIT^{tree}) \" \"$good_trees_file\")\"\n> +\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n> \tthen\n> \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n> \t\t# Continue the build job.\n> @@ -45,18 +66,18 @@ skip_good_tree () {\n> \techo \"$good_tree_info\" | {\n> \t\tread tree prev_good_commit prev_good_job_number prev_good_job_id\n> \n> -\t\tif test \"$TRAVIS_JOB_ID\" = \"$prev_good_job_id\"\n> +\t\tif test \"$CI_JOB_ID\" = \"$prev_good_job_id\"\n> \t\tthen\n> \t\t\tcat <<-EOF\n> -\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n> +\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n> \t\t\tThis commit has already been built and tested successfully by this build job.\n> \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n> \t\t\tEOF\n> \t\telse\n> \t\t\tcat <<-EOF\n> -\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n> +\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n> \t\t\tThis commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.\n> -\t\t\tThe log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id\n> +\t\t\tThe log of that build job is available at $(url_for_job_id $prev_good_job_id)\n> \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n> \t\t\tEOF\n> \t\tfi\n> @@ -81,7 +102,6 @@ check_unignored_build_artifacts ()\n> # and installing dependencies.\n> set -ex\n\n... after we turn on 'set -x', so the variables' values will be\nvisible in the logs.\n\n(Or move this 'set -ex' to the beginning of the script? Then we\ncould perhaps avoid similar issues in the future.)\n\n> -cache_dir=\"$HOME/travis-cache\"\n> good_trees_file=\"$cache_dir/good-trees\"\n> \n> mkdir -p \"$cache_dir\"\n"},{"id":"360892","messageId":"xmqqy3au3d26.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"20181018220106.GU19800@szeder.dev","subject":"Re: [PATCH v2 02/13] ci/lib.sh: encapsulate Travis-specific things","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-19T02:06:25Z","receivedAt":"2018-10-19T02:06:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n> On Mon, Oct 15, 2018 at 03:12:00AM -0700, Johannes Schindelin via GitGitGadget wrote:\n>> diff --git a/ci/lib.sh b/ci/lib.sh\n>> index 06970f7213..8532555b4e 100755\n>> --- a/ci/lib.sh\n>> +++ b/ci/lib.sh\n>> @@ -1,5 +1,26 @@\n>> # Library of functions shared by all CI scripts\n>> \n>> +if test true = \"$TRAVIS\"\n>> +then\n>> +...\n>> +\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n>> +\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n>> +fi\n>\n> Please set all these variables ...\n\nDo you mean \"VAR=VAL; export VAR\" is kosher, \"export VAR=VAL\" is\nnot?\n\n>> @@ -81,7 +102,6 @@ check_unignored_build_artifacts ()\n>> # and installing dependencies.\n>> set -ex\n>\n> ... after we turn on 'set -x', so the variables' values will be\n> visible in the logs.\n\nAh, no, you didn't. Although I think both are valid points, I think\nci/lib.sh is expected to be used only inside a more predictable\nenvironment (e.g. we know the shell used is not a random POSIX shell\nbut one that is happy with \"export VAR=VAL\"), so it should be OK.\nShowing the values of these variables in the log may still be good\nidea.\n\n> (Or move this 'set -ex' to the beginning of the script? Then we\n> could perhaps avoid similar issues in the future.)\n\nSure (provided that it is an issue to begin with---if we are\ninterested in the value of TRAVIS_BRANCH, for example, being able to\nsee it only because \"CI_BRANCH=$TRAVIS_BRANCH\" assignment is made\nfeels a bit sideways---we'd be better off explicitly logging\nanything we are interested in in the longer term, no?).\n"},{"id":"360905","messageId":"20181019082552.GV19800@szeder.dev","threadId":"49270","inReplyTo":"xmqqy3au3d26.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 02/13] ci/lib.sh: encapsulate Travis-specific things","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-19T08:25:52Z","receivedAt":"2018-10-19T08:25:59Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Oct 19, 2018 at 11:06:25AM +0900, Junio C Hamano wrote:\n> SZEDER Gábor writes:\n> \n> > On Mon, Oct 15, 2018 at 03:12:00AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> >> diff --git a/ci/lib.sh b/ci/lib.sh\n> >> index 06970f7213..8532555b4e 100755\n> >> --- a/ci/lib.sh\n> >> +++ b/ci/lib.sh\n> >> @@ -1,5 +1,26 @@\n> >> # Library of functions shared by all CI scripts\n> >> \n> >> +if test true = \"$TRAVIS\"\n> >> +then\n> >> +...\n> >> +\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n> >> +\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n> >> +fi\n> >\n> > Please set all these variables ...\n> \n> Do you mean \"VAR=VAL; export VAR\" is kosher, \"export VAR=VAL\" is\n> not?\n> \n> >> @@ -81,7 +102,6 @@ check_unignored_build_artifacts ()\n> >> # and installing dependencies.\n> >> set -ex\n> >\n> > ... after we turn on 'set -x', so the variables' values will be\n> > visible in the logs.\n> \n> Ah, no, you didn't. Although I think both are valid points, I think\n> ci/lib.sh is expected to be used only inside a more predictable\n> environment (e.g. we know the shell used is not a random POSIX shell\n> but one that is happy with \"export VAR=VAL\"), so it should be OK.\n\nYes. Travis CI runs an Ubuntu LTS, where /bin/sh is dash, which\nunderstands 'export VAR=val' just fine. I don't know what Linux\ndistro runs on Azure Pipelines, but the build definition in patch 6\nexplicitly asks for Bash, so that should be fine as well.\n\n> Showing the values of these variables in the log may still be good\n> idea.\n> \n> > (Or move this 'set -ex' to the beginning of the script? Then we\n> > could perhaps avoid similar issues in the future.)\n> \n> Sure (provided that it is an issue to begin with---if we are\n> interested in the value of TRAVIS_BRANCH, for example, being able to\n> see it only because \"CI_BRANCH=$TRAVIS_BRANCH\" assignment is made\n> feels a bit sideways---we'd be better off explicitly logging\n> anything we are interested in in the longer term, no?).\n\nWell, all the $TRAVIS_* and $CI_* variables are not that interessant,\nbecause they are only used in the build scripts, and then we can see\ntheir substituted values in the build logs. The same applies to\nthe variables $cache_dir and $BREW_INSTALL_PACKAGES as well.\n\n$GIT_PROVE_OPTS and $GIT_TEST_OPTS, however, are only used in\n't/Makefile' but not in the build scripts, thus their values don't\nshow up in the build logs.\n\nI run some Travis CI builds with custom $GIT_TEST_OPTS, which, of\ncourse, conflicted with this patch, and I messed up the conflict\nresolution. I think I would have noticed sooner what went wrong, if\nthe value of $GIT_TEST_OPTS were visible in the build logs.\n\nI've found the output with 'set -x' sufficient so far, and don't think\nthat an explicit logging facility is worth it.\n\n"},{"id":"360907","messageId":"xmqqva5yz64e.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"20181019082552.GV19800@szeder.dev","subject":"Re: [PATCH v2 02/13] ci/lib.sh: encapsulate Travis-specific things","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-19T08:35:13Z","receivedAt":"2018-10-19T08:35:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n>> >> +if test true = \"$TRAVIS\"\n>> >> +then\n>> >> +...\n>> >> +\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n>> >> +\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n>> >> +fi\n>> > ...\n>\n> $GIT_PROVE_OPTS and $GIT_TEST_OPTS, however, are only used in\n> 't/Makefile' but not in the build scripts, thus their values don't\n> show up in the build logs.\n>\n> I run some Travis CI builds with custom $GIT_TEST_OPTS, which, of\n> course, conflicted with this patch, and I messed up the conflict\n> resolution. I think I would have noticed sooner what went wrong, if\n> the value of $GIT_TEST_OPTS were visible in the build logs.\n>\n> I've found the output with 'set -x' sufficient so far, and don't think\n> that an explicit logging facility is worth it.\n\nThat's an interesting perspective. I would have thoguht that the\nvalues of these two variables we can see above, that are constants\nwithout any substitution or customization, are the least interesting\nthings to see in the log.\n\n\n"},{"id":"360910","messageId":"87k1mecj96.fsf@evledraar.gmail.com","threadId":"49270","inReplyTo":"xmqqo9bsvwn9.fsf@gitster-ct.c.googlers.com","subject":"Mirror of git.git on gitlab.com","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-10-19T10:40:05Z","receivedAt":"2018-10-19T10:40:10Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 18 2018, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>>> sites could do the same polling and mirroring. I am just too lazy\n>>> to open a new account at yet another hosting site to add that for\n>>> loop, but I may choose to when I am absolutely bored and nothing\n>>> else to do ;-).\n>>\n>> Do you mind if I squat gitlab.com/git/git in the meantime (i.e. create\n>> an org etc.) and have it mirror github.com/git/git?, I'll hand the\n>\n> Obviously somebody who is not even interested in obtaining an\n> account would appreciate, not just \"would not mind\", if a trusted\n> member in the community did that for the community ;-)\n\nI've set this up at https://gitlab.com/git-vcs\n\nThe /git namespace was taken (and I asked GitLab support if it was\nstale, they said no). Also tried /git-scm and /gitscm, ditto. So I\nsettled on /git-vcs (version control system).\n\nI mirrored all the repos that were on github.com/git (except for\ncabal.git of course). This is being kept up-to-date with GitLab's own\nmirroring feature, so it should always be max ~15m out of date v.s. the\nGitHub version.\n\nAs an aside, I noticed that\nhttps://github.com/git/sha1collisiondetection/ has never worked in\ncombination with git.git, i.e. it's cloned at a version that pre-dates\nthe initial introduction of the sha1collisiondetection submodule. Our\nother mirrors don't seem to have it at all relative to\n../sha1collisiondetection.git from their git.git.\n\nJunio: What was the plan with that? It's never been used in combination\nwith git.git, so maybe we should just drop it?\n\nIf we'd like to keep it and if it was kept up-to-date some scripts of\nyours we could use a relative URL in the .gitmodules, so e.g. if you\ncloned from kernel.org you'd also get the submodule from there.\n\nAlso since we're fixing it, it would be good to delete the existing repo\nand re-make it by clicking \"fork\" on\nhttps://github.com/cr-marcstevens/sha1collisiondetection so GitHub shows\n\"forked from...\", i.e. it's associated with the parent project in\nGitHub's UI.\n"},{"id":"361090","messageId":"20181021112053.GC30222@szeder.dev","threadId":"49270","inReplyTo":"12d6137f8d2ccc2041bed8d56d88a09b1db0fd77.1539598316.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 07/13] tests: introduce `test_atexit`","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-10-21T11:20:53Z","receivedAt":"2018-10-21T11:23:26Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Oct 15, 2018 at 03:12:08AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> From: Johannes Schindelin \n> \n> When running the p4 daemon or `git daemon`, we want to kill it at the\n> end of the test script.\n> \n> So far, we do this \"manually\".\n> \n> However, in the next few commits we want to teach the test suite to\n> optionally re-run scripts with different options, therefore we will have\n> to have a consistent way to stop daemons.\n> \n> Let's introduce `test_atexit`, which is loosely modeled after\n> `test_when_finished` (but has a broader scope: rather than running the\n> commands after the current test case, run them when the test script\n> finishes, and also run them when the `--immediate` option is in effect).\n\nI think killing daemons on failure even with '--immediate' will make\nlife a bit easier. I remember several occasions when I tried to debug\na failing test with '--immediate', some daemon stayed alive, and then\nthe next time I run the test it failed during setup.\n\nBesides 'git daemon' and P4, the httpd tests could benefit from this\nas well.\n\n\n> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\n> index 78d8c3783b..d7dd0c1be9 100644\n> --- a/t/test-lib-functions.sh\n> +++ b/t/test-lib-functions.sh\n> @@ -891,6 +891,35 @@ test_when_finished () {\n> \t\t} && (exit \\\"\\$eval_ret\\\"); eval_ret=\\$?; $test_cleanup\"\n> }\n> \n> +# This function can be used to schedule some commands to be run\n> +# unconditionally at the end of the test script, e.g. to stop a daemon:\n> +#\n> +#\ttest_expect_success 'test git daemon' '\n> +#\t\tgit daemon &\n> +#\t\tdaemon_pid=$! &&\n> +#\t\ttest_atexit \"kill $daemon_pid\" &&\n> +#\t\thello world\n> +#\t'\n\nI think we should add something like this as well:\n\n# Note that these commands will be run even when a test script run\n# with '--immediate' fails. Be careful with your commands to minimize\n# any changes to the failed state.\n\n> +\n> +test_atexit () {\n> +\t# We cannot detect when we are in a subshell in general, but by\n> +\t# doing so on Bash is better than nothing (the test will\n> +\t# silently pass on other shells).\n> +\ttest \"${BASH_SUBSHELL-0}\" = 0 ||\n> +\terror \"bug in test script: test_atexit does nothing in a subshell\"\n> +\ttest_atexit_cleanup=\"{ $*\n> +\t\t} && (exit \\\"\\$eval_ret\\\"); eval_ret=\\$?; $test_atexit_cleanup\"\n> +}\n> +\n> +test_atexit_handler () {\n> +\ttest : != \"$test_atexit_cleanup\" || return 0\n> +\n> +\tsetup_malloc_check\n> +\ttest_eval_ \"$test_atexit_cleanup\"\n> +\ttest_atexit_cleanup=:\n> +\tteardown_malloc_check\n> +}\n\nThe file 'test-lib-functions.sh' contains helper functions to be used\nin tests. 'test_atexit_handler' is not such a function, so I think it\nwould be better to add it to 'test-lib.sh'.\n\n> +\n> # Most tests can use the created repository, but some may need to create more.\n> # Usage: test_create_repo \n> test_create_repo () {\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 7ed0013f6d..6f9c1f5300 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -413,6 +413,7 @@ test_external_has_tap=0\n> \n> die () {\n> \tcode=$?\n\n# Run atexit handlers even when a test script run with\n# '--immediate' fails.\n\n> +\ttest_atexit_handler || code=$?\n> \tif test -n \"$GIT_EXIT_OK\"\n> \tthen\n> \t\texit $code\n> @@ -826,9 +827,12 @@ write_junit_xml_testcase () {\n> \tjunit_have_testcase=t\n> }\n> \n> +test_atexit_cleanup=:\n> test_done () {\n> \tGIT_EXIT_OK=t\n> \n> +\ttest -n \"$immediate\" || test_atexit_handler\n\nYou mentioned elsewhere [1] that you'll remove the condition to run\n'test_atexit_handler' unconditionally. I think this is the right\nthing to do, because this condition is responsible for the hanging P4\ntests I reported in [2].\n\nSo, 'lib-git-p4.sh' stores the pid of the 'p4d' process in the file\n\"$TRASH_DIRECTORY/p4d.pid\" in 'start_p4d', to be read later in\n'kill_p4d' to know which process to kill. When a test script run with\n'--immediate' ends successfully, and the atexit handlers are run in\n'die' above instead of in 'test_done', then you've got the problem\nthat 'test_done' removes the trash directory along with the 'p4d.pid'\nfile before the atexit handlers are run. With the pidfile gone\n'kill_p4d' won't know which process to kill, and the 'p4d' process\nwill stay alive.\n\nIf that condition is removed, then the atexit handlers will be run\nbefore the trash directory is removed, so 'p4d' processes will be\nstopped as they should even when an '--immediate' run succeeds. Good.\nNote, however, that in this case 'test_atexit_handler' will be called\ntwice: first in 'test_done' and then in 'die' as well. This should be\nfine, because 'test_atexit_handler' resets 'test_atexit_cleanup=:', so\nthe second call will do nothing. Great. Perhaps this would be worth\npointing out explicitly in the commit message and/or a comment.\n\n[1] https://public-inbox.org/git/nycvar.QRO.7.76.6.1810161111240.4546@tvgsbejvaqbjf.bet/\n[2] https://public-inbox.org/git/20181017232952.GT19800@szeder.dev/\n\n> +\n> \tif test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n> \tthen\n> \t\ttest -n \"$junit_have_testcase\" || {\n> -- \n> gitgitgadget\n\n"},{"id":"361138","messageId":"xmqq1s8iwnb4.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"87k1mecj96.fsf@evledraar.gmail.com","subject":"Re: Mirror of git.git on gitlab.com","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-22T05:41:19Z","receivedAt":"2018-10-22T05:41:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> I've set this up at https://gitlab.com/git-vcs\n>\n> The /git namespace was taken (and I asked GitLab support if it was\n> stale, they said no). Also tried /git-scm and /gitscm, ditto. So I\n> settled on /git-vcs (version control system).\n\nSquatters X-<. Thanks.\n\n> As an aside, I noticed that\n> https://github.com/git/sha1collisiondetection/ has never worked in\n> combination with git.git, i.e. it's cloned at a version that pre-dates\n> the initial introduction of the sha1collisiondetection submodule. Our\n> other mirrors don't seem to have it at all relative to\n> ../sha1collisiondetection.git from their git.git.\n\nI do not recall who cloned it or forked it there or what our longer\nterm plans for that repository would be. I think we are using this\nthing: https://github.com/cr-marcstevens/sha1collisiondetection.git\nand there is probably no reason to have our own copy. Perhaps we\nshould just get rid of it.\n\n\n"},{"id":"361486","messageId":"xmqqo9biigod.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-25T08:17:22Z","receivedAt":"2018-10-25T08:17:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> For a long time already, we have Git's source code continuously tested via\n> Travis CI, see e.g. https://travis-ci.org/git/git/builds/421738884. It has\n> served us well, and more and more developers actually pay attention and\n> benefit from the testing this gives us.\n\nWhat's the current status of this topic? Has the \"p4 daemon gets\nleft behind\" one resolved to everybody's satisfaction? I think that\none was the only large discussion on the series (aside from \"do we\nwant to keep Travis?\" subthread, which does not make this series\nundesirable), modulo your \"oy oy oy that is leftover debugging I\nneed to remove in a reroll\".\n\nThe topic was marked as \"On hold, monitoring discussion\" and it\nseems that discussion has quieted down, so the next step is to see\nan updated series?\n\nThanks.\n\n\n"},{"id":"361488","messageId":"20181025083750.GA16581@sigill.intra.peff.net","threadId":"49270","inReplyTo":"xmqq1s8iwnb4.fsf@gitster-ct.c.googlers.com","subject":"Re: Mirror of git.git on gitlab.com","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-10-25T08:37:50Z","receivedAt":"2018-10-25T08:37:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Oct 22, 2018 at 02:41:19PM +0900, Junio C Hamano wrote:\n\n> > As an aside, I noticed that\n> > https://github.com/git/sha1collisiondetection/ has never worked in\n> > combination with git.git, i.e. it's cloned at a version that pre-dates\n> > the initial introduction of the sha1collisiondetection submodule. Our\n> > other mirrors don't seem to have it at all relative to\n> > ../sha1collisiondetection.git from their git.git.\n> \n> I do not recall who cloned it or forked it there or what our longer\n> term plans for that repository would be. I think we are using this\n> thing: https://github.com/cr-marcstevens/sha1collisiondetection.git\n> and there is probably no reason to have our own copy. Perhaps we\n> should just get rid of it.\n\nI poked around. It was done by you on May 23, 2017 and then never\ntouched after that. I couldn't find any discussion on the list. So yeah,\nI think it makes sense to just delete it.\n\n-Peff\n"},{"id":"361611","messageId":"nycvar.QRO.7.76.6.1810261033440.4546@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqo9biigod.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2018-10-26T08:37:16Z","receivedAt":"2018-10-26T08:37:18Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Thu, 25 Oct 2018, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > For a long time already, we have Git's source code continuously tested via\n> > Travis CI, see e.g. https://travis-ci.org/git/git/builds/421738884. It has\n> > served us well, and more and more developers actually pay attention and\n> > benefit from the testing this gives us.\n> \n> What's the current status of this topic? Has the \"p4 daemon gets\n> left behind\" one resolved to everybody's satisfaction?\n\nNo. I was kind of waiting for Luke's answer, and in the alternative I\nhoped to find some time to work on trying to reproduce his issues on my\nsystem (but I failed to find said time so far).\n\n> I think that one was the only large discussion on the series (aside from\n> \"do we want to keep Travis?\" subthread, which does not make this series\n> undesirable), modulo your \"oy oy oy that is leftover debugging I need to\n> remove in a reroll\".\n> \n> The topic was marked as \"On hold, monitoring discussion\" and it\n> seems that discussion has quieted down, so the next step is to see\n> an updated series?\n\nI really think that I have to figure out what causes those p4d issues\nbefore I can give you that updated. I *am* interested, to be sure, it's\njust that other things seem to get in my way all the time.\n\nOne thing that keeps getting in my way, for example, is the performance\nissue identified in the chain linter. And I do think that I have to take\nthis into consideration for another update to this here patch series, too,\nby adding `--no-chain-lint` to the Windows phase. There is a similar thing\nwith `--with-dashes`, too.\n\nWill keep you updated,\nDscho\n\n> \n> Thanks.\n> \n> \n> \n"},{"id":"366826","messageId":"75ec97b3921f3ed346e9ab119ebff2546f03fade.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 01/21] travis: fix skipping tagged releases","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:11Z","receivedAt":"2019-01-16T13:36:15Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWhen building a PR, TRAVIS_BRANCH refers to the *target branch*.\nTherefore, if a PR targets `master`, and `master` happened to be tagged,\nwe skipped the build by mistake.\n\nFix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*)\nwhen available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also\nknown as \"push builds\").\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib-travisci.sh | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh\nindex 69dff4d1ec..d9d4f1a9d7 100755\n--- a/ci/lib-travisci.sh\n+++ b/ci/lib-travisci.sh\n@@ -1,5 +1,9 @@\n # Library of functions shared by all CI scripts\n \n+# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n+# want here. We want the source branch instead.\n+TRAVIS_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+\n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n \t# at the same commit as the tip of the branch is pushed, and building\n-- \ngitgitgadget\n\n"},{"id":"366827","messageId":"pull.31.v3.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v2.git.gitgitgadget@gmail.com","subject":"[PATCH v3 00/21] Offer to run CI/PR builds in Azure Pipelines","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:10Z","receivedAt":"2019-01-16T13:36:17Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"For a long time already, we have tested Git's source code continuously via\nTravis CI, see e.g. https://travis-ci.org/git/git/builds/421738884. It has\nserved us well, and more and more developers actually pay attention and\nbenefit from the testing this gives us.\n\nIt is also an invaluable tool for contributors who can validate their code\ncontributions via PRs on GitHub, e.g. to verify that their tests do actually\nrun on macOS (i.e. with the BSD family of Unix tools instead of the GNU\none).\n\nThe one sad part about this is the Windows support. Travis used to lack it\n(Travis announced some early support for Windows, hot off the press\n[https://blog.travis-ci.com/2018-10-11-windows-early-release]), and we work\naround that by using Azure Pipelines (the CI part of Azure DevOps, formerly\nknown as Visual Studio Team Services) indirectly: one phase in Travis would\ntrigger a build, wait for its log, and then paste that log.\n\nAs Git's Windows builds (and tests!) take quite a bit of time, Travis often\ntimed out, or somehow the trigger did not work, and for security reasons\n(the Windows builds are performed in a private pool of containers), the\nWindows builds are completely disabled for Pull Requests on GitHub.\n\nOne might ask why we did not use Azure Pipelines directly. There were a\ncouple of reasons for that:\n\n * most notably, Azure Pipelines' build logs could not be viewed\n anonymously,\n * while Azure Pipelines had Linux and Windows agents, it lacked macOS\n agents,\n * etc\n\nThe main two reasons no longer apply: macOS agents are available now\n[https://docs.microsoft.com/en-us/azure/devops/release-notes/2018/jul-10-vsts]\n, and are public projects\n[https://docs.microsoft.com/en-us/azure/devops/organizations/public/about-public-projects] \nnow, i.e. it is possible to configure a Azure Pipelines project so that \nanybody can view the logs. Since I offered v1, Azure Pipelines has been made\navailable via the GitHub Marketplace, free of cost for open source projects.\n\nI had secured such a public project for Git for Windows already, and I also\ngot one for Git. For now, the latter is hooked up with my personal git.git\nfork on GitHub, but it is my hope that I convince y'all that these Azure\nPipelines builds are a good idea, and then hook it up with \nhttps://github.com/git/git.\n\nAs a special treat, this patch series adds the ability to present the\noutcome of Git's test suite as JUnit-style .xml files. This allows the Azure\nPipelines build to present fun diagrams, trends, and makes it a lot easier\nto drill down to test failures than before. See for example \nhttps://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details\n[https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details] \n(you can click on the label of the failed test, and then see the detailed\noutput in the right pane).\n\nBut maybe you're not interested as much in the presentation of test failures\nas in the speed? Well, I got you covered with that, too. As of v3, the test\nsuite is run in parallel on Windows, cutting the overall run time to ~33\nminutes (see the latest build\n[https://dev.azure.com/git/git/_build/results?buildId=302&view=logs], for\nexample).\n\nThis patch series took waaaaaaaay more time than I had originally\nanticipated, but I think that in particular the advanced display of the test\nresults and the reduction of the overall run time was worth it. Please let\nme know what you think about this.\n\nChanges since v2:\n\n * Removed left-over debugging code that would skip a good chunk of \n t0000-init.sh.\n * Fixed the URL of the build badge.\n * Removed a trailing empty line from, and added a missing closing pointy\n bracket to, ci/mount-fileshare.sh.\n * Moved the \"travis: fix skipping tagged releases\" commit up to the\n beginning of the patch series.\n * The commit message of \"ci/lib.sh: add support for Azure Pipelines\" now\n mentions explicitly that the Homebrew packages that need to be installed\n on Travis' macOS agents are already installed on Azure Pipelines'.\n * Some commands were not guarded by || exit 1, i.e. if they would fail, the\n build would not have failed.\n * We now install gcc-8 for the linux-gcc job.\n * We no longer try to re-run failed tests with verbose log. Instead, we\n simply use the verbose log to begin with. Tests showed that it had a\n negligible impact on the overall run time.\n * The test_atexit_handler function was scratched; It would be the right\n thing to do, but is actually an independent topic (it was only\n implemented in v2 to accommodate the \"re-run with verbose log on failure\"\n trick)\n * We now use a new YAML schema (most notably, \"phase\" is now known as\n \"job\")\n * The Windows job contained PowerShell sections that were indented with 3\n spaces instead of 2.\n * The Windows job is now structured better, by separating different\n concerns into different \"tasks\" so that it is easier to see what exactly\n failed (was it the build? was it the test?)\n * The Windows job was split into a job to build Git and 10 parallel jobs to\n run the test suite with the artifacts built by the first job. This\n reduces the overall run time from ~1h20 (which was the run time by the\n Windows job) to ~35 minutes (which is the run time of the linux-gcc job).\n * The JUnit XML is now written using a test helper rather than a fragile\n and probably not even portable sed call.\n * Since we needed to determine the file size of the verbose log (to cut out\n individual test cases' log output), we now introduce a test helper to do\n that, and use it throughout the test suite (where Perl was used before).\n * It would appear that a recent change either in Cygwin or in the Azure VMs\n causes problems sporadically where the trash directories cannot be\n removed, but a subsequent rm will succeed. We now simply do that, because\n it won't harm the common case (where the first rm succeeds already) and\n because it helps the Windows job succeed pretty reliably.\n\nChanges since v1:\n\n * Removed a superfluous eval.\n * Added the commit that fixes the Travis PR builds targeting master that \n just happens to be tagged (see \n https://travis-ci.org/git/git/jobs/424276413 for an incorrectly-skipped\n build).\n * The commit messages and the cover letter now reflect the name change from\n Visual Studio Team Services to Azure DevOps (and in particular, Azure\n Pipelines for the automated builds).\n * Now we're using test_atexit (which we introduced for that purpose)\n instead of hard-coding kill_p4d and stop_git_daemon.\n * The build should now also succeed for Pull Requests (where secret\n variables are not available, for security reasons, and as a consequence\n the file share cannot be mounted).\n * The shell scripted parts now use proper && chains.\n\nJohannes Schindelin (21):\n travis: fix skipping tagged releases\n ci: rename the library of common functions\n ci/lib.sh: encapsulate Travis-specific things\n ci: inherit --jobs via MAKEFLAGS in run-build-and-tests\n ci: use a junction on Windows instead of a symlink\n test-date: add a subcommand to measure times in shell scripts\n tests: optionally write results as JUnit-style .xml\n ci/lib.sh: add support for Azure Pipelines\n Add a build definition for Azure DevOps\n ci: move the Windows job to the top\n ci: use git-sdk-64-minimal build artifact\n mingw: be more generous when wrapping up the setitimer() emulation\n README: add a build badge (status of the Azure Pipelines build)\n tests: avoid calling Perl just to determine file sizes\n tests: include detailed trace logs with --write-junit-xml upon failure\n mingw: try to work around issues with the test cleanup\n tests: add t/helper/ to the PATH with --with-dashes\n t0061: fix with --with-dashes and RUNTIME_PREFIX\n tests: optionally skip bin-wrappers/\n ci: speed up Windows phase\n ci: parallelize testing on Windows\n\n Makefile | 11 +\n README.md | 2 +\n azure-pipelines.yml | 344 ++++++++++++++++++++++++++++\n ci/install-dependencies.sh | 5 +-\n ci/{lib-travisci.sh => lib.sh} | 70 ++++--\n ci/make-test-artifacts.sh | 12 +\n ci/mount-fileshare.sh | 25 ++\n ci/print-test-failures.sh | 4 +-\n ci/run-build-and-tests.sh | 9 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/run-test-slice.sh | 17 ++\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 3 +-\n compat/mingw.c | 2 +-\n t/.gitignore | 1 +\n t/README | 9 +\n t/helper/test-date.c | 12 +\n t/helper/test-path-utils.c | 43 ++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/helper/test-xml-encode.c | 80 +++++++\n t/t0021-conversion.sh | 2 +-\n t/t0061-run-command.sh | 3 +-\n t/t1050-large.sh | 2 +-\n t/t5315-pack-objects-compression.sh | 2 +-\n t/t9303-fast-import-compression.sh | 2 +-\n t/test-lib.sh | 138 ++++++++++-\n 28 files changed, 768 insertions(+), 38 deletions(-)\n create mode 100644 azure-pipelines.yml\n rename ci/{lib-travisci.sh => lib.sh} (58%)\n create mode 100755 ci/make-test-artifacts.sh\n create mode 100755 ci/mount-fileshare.sh\n create mode 100755 ci/run-test-slice.sh\n create mode 100644 t/helper/test-xml-encode.c\n\n\nbase-commit: 77556354bb7ac50450e3b28999e3576969869068\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-31%2Fdscho%2Fvsts-ci-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-31/dscho/vsts-ci-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/31\n\nRange-diff vs v2:\n\n -: ---------- > 1: 75ec97b392 travis: fix skipping tagged releases\n 1: c963184510 = 2: d520f45108 ci: rename the library of common functions\n 2: 815152e0f5 ! 3: 06fa564386 ci/lib.sh: encapsulate Travis-specific things\n @@ -28,10 +28,14 @@\n @@\n # Library of functions shared by all CI scripts\n \n +-# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n +-# want here. We want the source branch instead.\n +-TRAVIS_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n +if test true = \"$TRAVIS\"\n +then\n -+\t# We are running within Travis CI\n -+\tCI_BRANCH=\"$TRAVIS_BRANCH\"\n ++\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n ++\t# what we want here. We want the source branch instead.\n ++\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n +\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n +\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n +\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n @@ -48,10 +52,9 @@\n +\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n +\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n +fi\n -+\n + \n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n - \t# at the same commit as the tip of the branch is pushed, and building\n @@\n \t# we can skip the build because we won't be skipping a build\n \t# of a tag.\n -: ---------- > 4: 52fb8e72fb ci: inherit --jobs via MAKEFLAGS in run-build-and-tests\n -: ---------- > 5: a35bc43a04 ci: use a junction on Windows instead of a symlink\n 3: 52337f1875 = 6: 16090ff67c test-date: add a subcommand to measure times in shell scripts\n 4: cf4c5ae470 ! 7: 272c0c0446 tests: optionally write results as JUnit-style .xml\n @@ -5,8 +5,26 @@\n This will come in handy when publishing the results of Git's test suite\n during an automated Azure DevOps run.\n \n + Note: we need to make extra sure that invalid UTF-8 encoding is turned\n + into valid UTF-8 (using the Replacement Character, \\uFFFD) because\n + t9902's trace contains such invalid byte sequences, and the task in the\n + Azure Pipeline that uploads the test results would refuse to do anything\n + if it was asked to parse an .xml file with invalid UTF-8 in it.\n +\n Signed-off-by: Johannes Schindelin \n \n + diff --git a/Makefile b/Makefile\n + --- a/Makefile\n + +++ b/Makefile\n +@@\n + TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o\n + TEST_BUILTINS_OBJS += test-subprocess.o\n + TEST_BUILTINS_OBJS += test-urlmatch-normalization.o\n ++TEST_BUILTINS_OBJS += test-xml-encode.o\n + TEST_BUILTINS_OBJS += test-wildmatch.o\n + TEST_BUILTINS_OBJS += test-windows-named-pipe.o\n + TEST_BUILTINS_OBJS += test-write-cache.o\n +\n diff --git a/t/.gitignore b/t/.gitignore\n --- a/t/.gitignore\n +++ b/t/.gitignore\n @@ -16,11 +34,121 @@\n /chainlinttmp\n +/out/\n \n + diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\n + --- a/t/helper/test-tool.c\n + +++ b/t/helper/test-tool.c\n +@@\n + \t{ \"submodule-nested-repo-config\", cmd__submodule_nested_repo_config },\n + \t{ \"subprocess\", cmd__subprocess },\n + \t{ \"urlmatch-normalization\", cmd__urlmatch_normalization },\n ++\t{ \"xml-encode\", cmd__xml_encode },\n + \t{ \"wildmatch\", cmd__wildmatch },\n + #ifdef GIT_WINDOWS_NATIVE\n + \t{ \"windows-named-pipe\", cmd__windows_named_pipe },\n +\n + diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\n + --- a/t/helper/test-tool.h\n + +++ b/t/helper/test-tool.h\n +@@\n + int cmd__submodule_nested_repo_config(int argc, const char **argv);\n + int cmd__subprocess(int argc, const char **argv);\n + int cmd__urlmatch_normalization(int argc, const char **argv);\n ++int cmd__xml_encode(int argc, const char **argv);\n + int cmd__wildmatch(int argc, const char **argv);\n + #ifdef GIT_WINDOWS_NATIVE\n + int cmd__windows_named_pipe(int argc, const char **argv);\n +\n + diff --git a/t/helper/test-xml-encode.c b/t/helper/test-xml-encode.c\n + new file mode 100644\n + --- /dev/null\n + +++ b/t/helper/test-xml-encode.c\n +@@\n ++#include \"test-tool.h\"\n ++\n ++static const char *utf8_replace_character = \"�\";\n ++\n ++/*\n ++ * Encodes (possibly incorrect) UTF-8 on to , to be embedded\n ++ * in an XML file.\n ++ */\n ++int cmd__xml_encode(int argc, const char **argv)\n ++{\n ++\tunsigned char buf[1024], tmp[4], *tmp2 = NULL;\n ++\tssize_t cur = 0, len = 1, remaining = 0;\n ++\tunsigned char ch;\n ++\n ++\tfor (;;) {\n ++\t\tif (++cur == len) {\n ++\t\t\tlen = xread(0, buf, sizeof(buf));\n ++\t\t\tif (!len)\n ++\t\t\t\treturn 0;\n ++\t\t\tif (len < 0)\n ++\t\t\t\tdie_errno(\"Could not read \");\n ++\t\t\tcur = 0;\n ++\t\t}\n ++\t\tch = buf[cur];\n ++\n ++\t\tif (tmp2) {\n ++\t\t\tif ((ch & 0xc0) != 0x80) {\n ++\t\t\t\tfputs(utf8_replace_character, stdout);\n ++\t\t\t\ttmp2 = 0;\n ++\t\t\t\tcur--;\n ++\t\t\t\tcontinue;\n ++\t\t\t}\n ++\t\t\t*tmp2 = ch;\n ++\t\t\ttmp2++;\n ++\t\t\tif (--remaining == 0) {\n ++\t\t\t\tfwrite(tmp, tmp2 - tmp, 1, stdout);\n ++\t\t\t\ttmp2 = 0;\n ++\t\t\t}\n ++\t\t\tcontinue;\n ++\t\t}\n ++\n ++\t\tif (!(ch & 0x80)) {\n ++\t\t\t/* 0xxxxxxx */\n ++\t\t\tif (ch == '&')\n ++\t\t\t\tfputs(\"&\", stdout);\n ++\t\t\telse if (ch == '\\'')\n ++\t\t\t\tfputs(\"'\", stdout);\n ++\t\t\telse if (ch == '\"')\n ++\t\t\t\tfputs(\""\", stdout);\n ++\t\t\telse if (ch == '<')\n ++\t\t\t\tfputs(\"<\", stdout);\n ++\t\t\telse if (ch == '>')\n ++\t\t\t\tfputs(\">\", stdout);\n ++\t\t\telse if (ch >= 0x20)\n ++\t\t\t\tfputc(ch, stdout);\n ++\t\t\telse if (ch == 0x09 || ch == 0x0a || ch == 0x0d)\n ++\t\t\t\tfprintf(stdout, \"&#x%02x;\", ch);\n ++\t\t\telse\n ++\t\t\t\tfputs(utf8_replace_character, stdout);\n ++\t\t} else if ((ch & 0xe0) == 0xc0) {\n ++\t\t\t/* 110XXXXx 10xxxxxx */\n ++\t\t\ttmp[0] = ch;\n ++\t\t\tremaining = 1;\n ++\t\t\ttmp2 = tmp + 1;\n ++\t\t} else if ((ch & 0xf0) == 0xe0) {\n ++\t\t\t/* 1110XXXX 10Xxxxxx 10xxxxxx */\n ++\t\t\ttmp[0] = ch;\n ++\t\t\tremaining = 2;\n ++\t\t\ttmp2 = tmp + 1;\n ++\t\t} else if ((ch & 0xf8) == 0xf0) {\n ++\t\t\t/* 11110XXX 10XXxxxx 10xxxxxx 10xxxxxx */\n ++\t\t\ttmp[0] = ch;\n ++\t\t\tremaining = 3;\n ++\t\t\ttmp2 = tmp + 1;\n ++\t\t} else\n ++\t\t\tfputs(utf8_replace_character, stdout);\n ++\t}\n ++\n ++\treturn 0;\n ++}\n +\n diff --git a/t/test-lib.sh b/t/test-lib.sh\n --- a/t/test-lib.sh\n +++ b/t/test-lib.sh\n @@\n - \t--verbose-log)\n + \t-V|--verbose-log)\n \t\tverbose_log=t\n \t\tshift ;;\n +\t--write-junit-xml)\n @@ -115,14 +243,7 @@\n +}\n +\n +xml_attr_encode () {\n -+\t# We do not translate CR to because BSD sed does not handle\n -+\t# \\r in the regex. In practice, the output should not even have any\n -+\t# carriage returns.\n -+\tprintf '%s\\n' \"$@\" |\n -+\tsed -e 's/&/\\&/g' -e \"s/'/\\'/g\" -e 's/\"/\\"/g' \\\n -+\t\t-e 's//\\>/g' \\\n -+\t\t-e 's/\t/\\ /g' -e 's/$/\\ /' -e '$s/ $//' |\n -+\ttr -d '\\012\\015'\n ++\tprintf '%s\\n' \"$@\" | test-tool xml-encode\n +}\n +\n +write_junit_xml_testcase () {\n 5: 486d1d2518 ! 8: fefab79d46 ci/lib.sh: add support for Azure Pipelines\n @@ -6,6 +6,10 @@\n variables and a function that displays the URL given the job id (to\n identify previous runs for known-good trees).\n \n + For example, we do not have to install the git-lfs and gettext packages\n + on Azure Pipelines' macOS agents: they are already installed, and trying\n + to install them again would result in an error.\n +\n Signed-off-by: Johannes Schindelin \n \n diff --git a/ci/lib.sh b/ci/lib.sh\n @@ -15,7 +19,7 @@\n \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n -+elif test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n ++elif test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n +then\n +\t# We are running in Azure Pipelines\n +\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n 6: 1a22efe849 ! 9: ddee8ecd8a Add a build definition for Azure DevOps\n @@ -28,12 +28,11 @@\n +- repo: self\n + fetchDepth: 1\n +\n -+phases:\n -+- phase: linux_clang\n ++jobs:\n ++- job: linux_clang\n + displayName: linux-clang\n + condition: succeeded()\n -+ queue:\n -+ name: Hosted Ubuntu 1604\n ++ pool: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n @@ -43,7 +42,7 @@\n +\n + export CC=clang || exit 1\n +\n -+ ci/install-dependencies.sh\n ++ ci/install-dependencies.sh || exit 1\n + ci/run-build-and-tests.sh || {\n + ci/print-test-failures.sh\n + exit 1\n @@ -62,19 +61,19 @@\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n +\n -+- phase: linux_gcc\n ++- job: linux_gcc\n + displayName: linux-gcc\n + condition: succeeded()\n -+ queue:\n -+ name: Hosted Ubuntu 1604\n ++ pool: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n ++ sudo add-apt-repository ppa:ubuntu-toolchain-r/test &&\n + sudo apt-get update &&\n -+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin || exit 1\n ++ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2 language-pack-is git-svn gcc-8 || exit 1\n +\n -+ ci/install-dependencies.sh\n ++ ci/install-dependencies.sh || exit 1\n + ci/run-build-and-tests.sh || {\n + ci/print-test-failures.sh\n + exit 1\n @@ -93,18 +92,17 @@\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n +\n -+- phase: osx_clang\n ++- job: osx_clang\n + displayName: osx-clang\n + condition: succeeded()\n -+ queue:\n -+ name: Hosted macOS\n ++ pool: Hosted macOS\n + steps:\n + - bash: |\n + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n + export CC=clang\n +\n -+ ci/install-dependencies.sh\n ++ ci/install-dependencies.sh || exit 1\n + ci/run-build-and-tests.sh || {\n + ci/print-test-failures.sh\n + exit 1\n @@ -123,16 +121,15 @@\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n +\n -+- phase: osx_gcc\n ++- job: osx_gcc\n + displayName: osx-gcc\n + condition: succeeded()\n -+ queue:\n -+ name: Hosted macOS\n ++ pool: Hosted macOS\n + steps:\n + - bash: |\n + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n -+ ci/install-dependencies.sh\n ++ ci/install-dependencies.sh || exit 1\n + ci/run-build-and-tests.sh || {\n + ci/print-test-failures.sh\n + exit 1\n @@ -151,11 +148,10 @@\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n +\n -+- phase: gettext_poison\n ++- job: gettext_poison\n + displayName: GETTEXT_POISON\n + condition: succeeded()\n -+ queue:\n -+ name: Hosted Ubuntu 1604\n ++ pool: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n @@ -183,73 +179,81 @@\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n +\n -+- phase: windows\n ++- job: windows\n + displayName: Windows\n + condition: succeeded()\n -+ queue:\n -+ name: Hosted VS2017\n -+ timeoutInMinutes: 240\n ++ pool: Hosted\n ++ timeoutInMinutes: 240\n + steps:\n + - powershell: |\n -+ # Helper to check the error level of the latest command (exit with error when appropriate)\n -+ function c() { if (!$?) { exit(1) } }\n -+\n -+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n -+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no; c\n -+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\; c\n -+ }\n -+\n -+ # Add build agent's MinGit to PATH\n -+ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n -+\n -+ # Helper to initialize (or update) a Git worktree\n -+ function init ($path, $url, $set_origin) {\n -+ if (Test-Path $path) {\n -+ cd $path; c\n -+ if (Test-Path .git) {\n -+ git init; c\n -+ } else {\n -+ git status\n -+ }\n -+ } else {\n -+ git init $path; c\n -+ cd $path; c\n -+ }\n -+ git config core.autocrlf false; c\n -+ git config core.untrackedCache true; c\n -+ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n -+ git remote add origin $url; c\n -+ }\n -+ git fetch --depth=1 $url master; c\n -+ git reset --hard FETCH_HEAD; c\n -+ git clean -df; c\n -+ }\n -+\n -+ # Initialize Git for Windows' SDK\n -+ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n -+ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n -+ init usr\\src\\build-extra https://github.com/git-for-windows/build-extra 1\n -+\n -+ cd \"$(Build.SourcesDirectory)\"; c\n -+\n -+ $env:HOME = \"$(Build.SourcesDirectory)\"\n -+ $env:MSYSTEM = \"MINGW64\"\n -+ git-sdk-64\\git-cmd --command=usr\\\\bin\\\\bash.exe -lc @\"\n -+ . ci/lib.sh\n -+\n -+ make -j10 DEVELOPER=1 NO_PERL=1 || exit 1\n -+ NO_PERL=1 NO_SVN_TESTS=1 GIT_TEST_OPTS=\\\"--quiet --write-junit-xml\\\" time make -j15 -k DEVELOPER=1 test || {\n -+ NO_PERL=1 NO_SVN_TESTS=1 GIT_TEST_OPTS=\\\"-i -v -x\\\" make -k -C t failed; exit 1\n -+ }\n -+\n -+ save_good_tree\n -+ \"@\n -+ c\n -+\n -+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n -+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n -+ }\n -+ displayName: 'build & test'\n ++ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n ++ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n ++ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n ++ }\n ++ displayName: 'Mount test-cache'\n ++ env:\n ++ GITFILESHAREPWD: $(gitfileshare.pwd)\n ++ - powershell: |\n ++ # Helper to check the error level of the latest command (exit with error when appropriate)\n ++ function c() { if (!$?) { exit(1) } }\n ++\n ++ # Add build agent's MinGit to PATH\n ++ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n ++\n ++ # Helper to initialize (or update) a Git worktree\n ++ function init ($path, $url, $set_origin) {\n ++ if (Test-Path $path) {\n ++ cd $path; c\n ++ if (Test-Path .git) {\n ++ & git init; c\n ++ } else {\n ++ & git status\n ++ }\n ++ } else {\n ++ & git init $path; c\n ++ cd $path; c\n ++ }\n ++ & git config core.autocrlf false; c\n ++ & git config core.untrackedCache true; c\n ++ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n ++ & git remote add origin $url; c\n ++ }\n ++ & git fetch --depth=1 $url master; c\n ++ & git reset --hard FETCH_HEAD; c\n ++ & git clean -df; c\n ++ }\n ++\n ++ # Initialize Git for Windows' SDK\n ++ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n ++ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n ++\n ++ # Let Git ignore the SDK and the test-cache\n ++ \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n ++ displayName: 'Initialize the Git for Windows SDK'\n ++ - powershell: |\n ++ & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n ++ export MAKEFLAGS=-j10\n ++ export DEVELOPER=1\n ++ export NO_PERL=1\n ++ export NO_SVN_TESTS=1\n ++ export GIT_TEST_SKIP_REBASE_P=1\n ++\n ++ ci/run-build-and-tests.sh || {\n ++ ci/print-test-failures.sh\n ++ exit 1\n ++ }\n ++ \"@\n ++ if (!$?) { exit(1) }\n ++ displayName: 'Build & Test'\n ++ env:\n ++ HOME: $(Build.SourcesDirectory)\n ++ MSYSTEM: MINGW64\n ++ - powershell: |\n ++ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n ++ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n ++ }\n ++ displayName: 'Unmount test-cache'\n ++ condition: true\n + env:\n + GITFILESHAREPWD: $(gitfileshare.pwd)\n + - task: PublishTestResults@2\n @@ -261,29 +265,14 @@\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n +\n -+- phase: linux32\n ++- job: linux32\n + displayName: Linux32\n + condition: succeeded()\n -+ queue:\n -+ name: Hosted Ubuntu 1604\n ++ pool: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n -+ sudo apt-get update &&\n -+ sudo apt-get -y install \\\n -+ apt-transport-https \\\n -+ ca-certificates \\\n -+ curl \\\n -+ software-properties-common &&\n -+ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - &&\n -+ sudo add-apt-repository \\\n -+ \"deb [arch=amd64] https://download.docker.com/linux/ubuntu \\\n -+ $(lsb_release -cs) \\\n -+ stable\" &&\n -+ sudo apt-get update &&\n -+ sudo apt-get -y install docker-ce &&\n -+\n + sudo AGENT_OS=\"$AGENT_OS\" BUILD_BUILDNUMBER=\"$BUILD_BUILDNUMBER\" BUILD_REPOSITORY_URI=\"$BUILD_REPOSITORY_URI\" BUILD_SOURCEBRANCH=\"$BUILD_SOURCEBRANCH\" BUILD_SOURCEVERSION=\"$BUILD_SOURCEVERSION\" SYSTEM_PHASENAME=\"$SYSTEM_PHASENAME\" SYSTEM_TASKDEFINITIONSURI=\"$SYSTEM_TASKDEFINITIONSURI\" SYSTEM_TEAMPROJECT=\"$SYSTEM_TEAMPROJECT\" CC=$CC MAKEFLAGS=-j3 bash -lxc ci/run-linux32-docker.sh || exit 1\n +\n + sudo chmod a+r t/out/TEST-*.xml\n @@ -301,11 +290,10 @@\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n +\n -+- phase: static_analysis\n ++- job: static_analysis\n + displayName: StaticAnalysis\n + condition: succeeded()\n -+ queue:\n -+ name: Hosted Ubuntu 1604\n ++ pool: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n @@ -322,11 +310,10 @@\n + env:\n + GITFILESHAREPWD: $(gitfileshare.pwd)\n +\n -+- phase: documentation\n ++- job: documentation\n + displayName: Documentation\n + condition: succeeded()\n -+ queue:\n -+ name: Hosted Ubuntu 1604\n ++ pool: Hosted Ubuntu 1604\n + steps:\n + - bash: |\n + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n @@ -344,6 +331,19 @@\n + env:\n + GITFILESHAREPWD: $(gitfileshare.pwd)\n \n + diff --git a/ci/lib.sh b/ci/lib.sh\n + --- a/ci/lib.sh\n + +++ b/ci/lib.sh\n +@@\n + \n + \tBREW_INSTALL_PACKAGES=\n + \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n +-\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n ++\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n + fi\n + \n + skip_branch_tip_with_tag () {\n +\n diff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\n new file mode 100755\n --- /dev/null\n @@ -357,7 +357,7 @@\n +}\n +\n +test $# = 4 ||\n -+die \"Usage: $0 \"\n +\n +mkdir -p \"$4\" || die \"Could not create $4\"\n +\n @@ -374,4 +374,3 @@\n +\t;;\n +esac ||\n +die \"Could not mount $4\"\n -+\n 7: 12d6137f8d < -: ---------- tests: introduce `test_atexit`\n 8: 3bb226b79b < -: ---------- git-daemon: use `test_atexit` in the tests\n 9: 3e2193a73d < -: ---------- git-p4: use `test_atexit` to kill the daemon\n 10: ae3c42519a < -: ---------- tests: include detailed trace logs with --write-junit-xml upon failure\n 11: 2466a48aa3 < -: ---------- tests: record more stderr with --write-junit-xml in case of failure\n -: ---------- > 10: 58bb8d0469 ci: move the Windows job to the top\n -: ---------- > 11: b6bb6a87a3 ci: use git-sdk-64-minimal build artifact\n -: ---------- > 12: 2b6ee78f02 mingw: be more generous when wrapping up the setitimer() emulation\n 12: d112b3fe86 ! 13: df326039d9 README: add a build badge (status of the Azure Pipelines build)\n @@ -10,7 +10,7 @@\n --- a/README.md\n +++ b/README.md\n @@\n -+[![Build Status](https:/dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2)\n ++[![Build Status](https://dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2)\n +\n Git - fast, scalable, distributed revision control system\n =========================================================\n 13: 0a53f37135 < -: ---------- travis: fix skipping tagged releases\n -: ---------- > 14: 0d547db8f7 tests: avoid calling Perl just to determine file sizes\n -: ---------- > 15: f678b105f8 tests: include detailed trace logs with --write-junit-xml upon failure\n -: ---------- > 16: 8ef674a236 mingw: try to work around issues with the test cleanup\n -: ---------- > 17: b503167084 tests: add t/helper/ to the PATH with --with-dashes\n -: ---------- > 18: 713910e1dc t0061: fix with --with-dashes and RUNTIME_PREFIX\n -: ---------- > 19: 3a77eafb44 tests: optionally skip bin-wrappers/\n -: ---------- > 20: fba97133bf ci: speed up Windows phase\n -: ---------- > 21: e568349930 ci: parallelize testing on Windows\n\n-- \ngitgitgadget\n"},{"id":"366828","messageId":"d520f451080c8e43e3159665ca62a7ab9b8ca445.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 02/21] ci: rename the library of common functions","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:12Z","receivedAt":"2019-01-16T13:36:18Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe name is hard-coded to reflect that we use Travis CI for continuous\ntesting.\n\nIn the next commits, we will extend this to be able use Azure DevOps,\ntoo.\n\nSo let's adjust the name to make it more generic.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 2 +-\n ci/{lib-travisci.sh => lib.sh} | 0\n ci/print-test-failures.sh | 2 +-\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 2 +-\n 8 files changed, 7 insertions(+), 7 deletions(-)\n rename ci/{lib-travisci.sh => lib.sh} (100%)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 06c3546e1e..fe65144152 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -3,7 +3,7 @@\n # Install dependencies required to build and test Git on Linux and macOS\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\ndiff --git a/ci/lib-travisci.sh b/ci/lib.sh\nsimilarity index 100%\nrename from ci/lib-travisci.sh\nrename to ci/lib.sh\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex d55460a212..7aef39a2fd 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -3,7 +3,7 @@\n # Print output of failing tests\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n # Tracing executed commands would produce too much noise in the loop below.\n set +x\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex cda170d5c2..db342bb6a8 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -3,7 +3,7 @@\n # Build and test Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n ln -s \"$cache_dir/.prove\" t/.prove\n \ndiff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh\nindex 21637903ce..751acfcf8a 100755\n--- a/ci/run-linux32-docker.sh\n+++ b/ci/run-linux32-docker.sh\n@@ -3,7 +3,7 @@\n # Download and run Docker image to build and test 32-bit Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n docker pull daald/ubuntu32:xenial\n \ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\nindex 5688f261d0..dc189c7456 100755\n--- a/ci/run-static-analysis.sh\n+++ b/ci/run-static-analysis.sh\n@@ -3,7 +3,7 @@\n # Perform various static code analysis checks\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n make --jobs=2 coccicheck\n \ndiff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh\nindex d99a180e52..a73a4eca0a 100755\n--- a/ci/run-windows-build.sh\n+++ b/ci/run-windows-build.sh\n@@ -6,7 +6,7 @@\n # supported) and a commit hash.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n test $# -ne 2 && echo \"Unexpected number of parameters\" && exit 1\n test -z \"$GFW_CI_TOKEN\" && echo \"GFW_CI_TOKEN not defined\" && exit\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex a20de9ca12..d3cdbac73f 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -3,7 +3,7 @@\n # Perform sanity checks on documentation and build it.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n gem install asciidoctor\n \n-- \ngitgitgadget\n\n"},{"id":"366829","messageId":"06fa564386307146762d4b30259d8636fc0ac268.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:13Z","receivedAt":"2019-01-16T13:36:19Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe upcoming patches will allow building git.git via Azure Pipelines\n(i.e. Azure DevOps' Continuous Integration), where variable names and\nURLs look a bit different than in Travis CI.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 3 ++-\n ci/lib.sh | 47 +++++++++++++++++++++++++-------------\n ci/print-test-failures.sh | 2 +-\n ci/test-documentation.sh | 1 +\n 4 files changed, 35 insertions(+), 18 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex fe65144152..bcdcc71592 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -37,7 +37,8 @@ osx-clang|osx-gcc)\n \tbrew update --quiet\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\n-\tbrew install git-lfs gettext\n+\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n+\tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \tbrew install caskroom/cask/perforce\n \t;;\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex d9d4f1a9d7..bfc406a63b 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,8 +1,26 @@\n # Library of functions shared by all CI scripts\n \n-# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n-# want here. We want the source branch instead.\n-TRAVIS_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+if test true = \"$TRAVIS\"\n+then\n+\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n+\t# what we want here. We want the source branch instead.\n+\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n+\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n+\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n+\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n+\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n+\n+\tcache_dir=\"$HOME/travis-cache\"\n+\n+\turl_for_job_id () {\n+\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+fi\n \n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n@@ -17,10 +35,10 @@ skip_branch_tip_with_tag () {\n \t# we can skip the build because we won't be skipping a build\n \t# of a tag.\n \n-\tif TAG=$(git describe --exact-match \"$TRAVIS_BRANCH\" 2>/dev/null) &&\n-\t\ttest \"$TAG\" != \"$TRAVIS_BRANCH\"\n+\tif TAG=$(git describe --exact-match \"$CI_BRANCH\" 2>/dev/null) &&\n+\t\ttest \"$TAG\" != \"$CI_BRANCH\"\n \tthen\n-\t\techo \"$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)\"\n+\t\techo \"$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)\"\n \t\texit 0\n \tfi\n }\n@@ -29,7 +47,7 @@ skip_branch_tip_with_tag () {\n # job if we encounter the same tree again and can provide a useful info\n # message.\n save_good_tree () {\n-\techo \"$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID\" >>\"$good_trees_file\"\n+\techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n \t# limit the file size\n \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n \tmv \"$good_trees_file\".tmp \"$good_trees_file\"\n@@ -39,7 +57,7 @@ save_good_tree () {\n # successfully before (e.g. because the branch got rebased, changing only\n # the commit messages).\n skip_good_tree () {\n-\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $TRAVIS_COMMIT^{tree}) \" \"$good_trees_file\")\"\n+\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n \tthen\n \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n \t\t# Continue the build job.\n@@ -49,18 +67,18 @@ skip_good_tree () {\n \techo \"$good_tree_info\" | {\n \t\tread tree prev_good_commit prev_good_job_number prev_good_job_id\n \n-\t\tif test \"$TRAVIS_JOB_ID\" = \"$prev_good_job_id\"\n+\t\tif test \"$CI_JOB_ID\" = \"$prev_good_job_id\"\n \t\tthen\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit has already been built and tested successfully by this build job.\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\telse\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.\n-\t\t\tThe log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id\n+\t\t\tThe log of that build job is available at $(url_for_job_id $prev_good_job_id)\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\tfi\n@@ -85,7 +103,6 @@ check_unignored_build_artifacts ()\n # and installing dependencies.\n set -ex\n \n-cache_dir=\"$HOME/travis-cache\"\n good_trees_file=\"$cache_dir/good-trees\"\n \n mkdir -p \"$cache_dir\"\n@@ -95,13 +112,11 @@ skip_good_tree\n \n if test -z \"$jobname\"\n then\n-\tjobname=\"$TRAVIS_OS_NAME-$CC\"\n+\tjobname=\"$CI_OS_NAME-$CC\"\n fi\n \n export DEVELOPER=1\n export DEFAULT_TEST_TARGET=prove\n-export GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n-export GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n export GIT_TEST_CLONE_2GB=YesPlease\n if [ \"$jobname\" = linux-gcc ]; then\n \texport CC=gcc-8\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 7aef39a2fd..d2045b63a6 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -69,7 +69,7 @@ do\n \tfi\n done\n \n-if [ $combined_trash_size -gt 0 ]\n+if [ -n \"$TRAVIS_JOB_ID\" -a $combined_trash_size -gt 0 ]\n then\n \techo \"------------------------------------------------------------------------\"\n \techo \"Trash directories embedded in this log can be extracted by running:\"\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex d3cdbac73f..7d0beb2832 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -5,6 +5,7 @@\n \n . ${0%/*}/lib.sh\n \n+test -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n gem install asciidoctor\n \n make check-builtins\n-- \ngitgitgadget\n\n"},{"id":"366830","messageId":"52fb8e72fb09a5b02e19de172ee28684d35af0a6.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 04/21] ci: inherit --jobs via MAKEFLAGS in run-build-and-tests","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:14Z","receivedAt":"2019-01-16T13:36:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nLet's not decide in the generic ci/ script how many jobs to run in\nparallel; it is easy enough to hand that information down via the\n`MAKEFLAGS`.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/run-build-and-tests.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex db342bb6a8..80d72d120f 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -7,7 +7,7 @@\n \n ln -s \"$cache_dir/.prove\" t/.prove\n \n-make --jobs=2\n+make\n make --quiet test\n if test \"$jobname\" = \"linux-gcc\"\n then\n-- \ngitgitgadget\n\n"},{"id":"366831","messageId":"16090ff67caeffe59bdc2294a93af634eb23e33e.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 06/21] test-date: add a subcommand to measure times in shell scripts","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:16Z","receivedAt":"2019-01-16T13:36:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIn the next commit, we want to teach Git's test suite to optionally\noutput test results in JUnit-style .xml files. These files contain\ninformation about the time spent. So we need a way to measure time.\n\nWhile we could use `date +%s` for that, this will give us only seconds,\ni.e. very coarse-grained timings.\n\nGNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output),\nbut there is no equivalent in BSD `date` (read: on macOS, we would not\nbe able to obtain precise timings).\n\nSo let's introduce `test-tool date getnanos`, with an optional start\ntime, that outputs preciser values.\n\nGranted, it is a bit pointless to try measuring times accurately in\nshell scripts, certainly to nanosecond precision. But it is better than\nsecond-granularity.\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-date.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex a0837371ab..792a805374 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -7,6 +7,7 @@ static const char *usage_msg = \"\\n\"\n \" test-tool date parse [date]...\\n\"\n \" test-tool date approxidate [date]...\\n\"\n \" test-tool date timestamp [date]...\\n\"\n+\" test-tool date getnanos [start-nanos]\\n\"\n \" test-tool date is64bit\\n\"\n \" test-tool date time_t-is64bit\\n\";\n \n@@ -82,6 +83,15 @@ static void parse_approx_timestamp(const char **argv, struct timeval *now)\n \t}\n }\n \n+static void getnanos(const char **argv, struct timeval *now)\n+{\n+\tdouble seconds = getnanotime() / 1.0e9;\n+\n+\tif (*argv)\n+\t\tseconds -= strtod(*argv, NULL);\n+\tprintf(\"%lf\\n\", seconds);\n+}\n+\n int cmd__date(int argc, const char **argv)\n {\n \tstruct timeval now;\n@@ -108,6 +118,8 @@ int cmd__date(int argc, const char **argv)\n \t\tparse_approxidate(argv+1, &now);\n \telse if (!strcmp(*argv, \"timestamp\"))\n \t\tparse_approx_timestamp(argv+1, &now);\n+\telse if (!strcmp(*argv, \"getnanos\"))\n+\t\tgetnanos(argv+1, &now);\n \telse if (!strcmp(*argv, \"is64bit\"))\n \t\treturn sizeof(timestamp_t) == 8 ? 0 : 1;\n \telse if (!strcmp(*argv, \"time_t-is64bit\"))\n-- \ngitgitgadget\n\n"},{"id":"366833","messageId":"58bb8d0469a4a9f7bd422b7ca60df4ab70ef5a29.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 10/21] ci: move the Windows job to the top","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:20Z","receivedAt":"2019-01-16T13:36:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe Windows job currently takes a whopping ~1h20m to complete. Which is\n*far* longer than the next-longest job takes (linux-gcc, ~35m). As such,\nit makes sense to start the Windows job first, to minimize the overall\nrun time (which is now pretty safely the run time of the Windows job).\n\nThis affects only the Azure Pipelines configuration, not the Travis one,\nof course, as Travis cannot run our Windows job: 1h20m is distinctly\nlonger than the 50 minute timeout of Travis' free tier.\n\nThis commit is best viewed with `--color-moved`.\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 172 ++++++++++++++++++++++----------------------\n 1 file changed, 86 insertions(+), 86 deletions(-)\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex f3cabb0dd0..e44d2733a4 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -3,6 +3,92 @@ resources:\n fetchDepth: 1\n \n jobs:\n+- job: windows\n+ displayName: Windows\n+ condition: succeeded()\n+ pool: Hosted\n+ timeoutInMinutes: 240\n+ steps:\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n+ }\n+ displayName: 'Mount test-cache'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - powershell: |\n+ # Helper to check the error level of the latest command (exit with error when appropriate)\n+ function c() { if (!$?) { exit(1) } }\n+\n+ # Add build agent's MinGit to PATH\n+ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n+\n+ # Helper to initialize (or update) a Git worktree\n+ function init ($path, $url, $set_origin) {\n+ if (Test-Path $path) {\n+ cd $path; c\n+ if (Test-Path .git) {\n+ & git init; c\n+ } else {\n+ & git status\n+ }\n+ } else {\n+ & git init $path; c\n+ cd $path; c\n+ }\n+ & git config core.autocrlf false; c\n+ & git config core.untrackedCache true; c\n+ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n+ & git remote add origin $url; c\n+ }\n+ & git fetch --depth=1 $url master; c\n+ & git reset --hard FETCH_HEAD; c\n+ & git clean -df; c\n+ }\n+\n+ # Initialize Git for Windows' SDK\n+ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n+ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+\n+ # Let Git ignore the SDK and the test-cache\n+ \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n+ displayName: 'Initialize the Git for Windows SDK'\n+ - powershell: |\n+ & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ export MAKEFLAGS=-j10\n+ export DEVELOPER=1\n+ export NO_PERL=1\n+ export NO_SVN_TESTS=1\n+ export GIT_TEST_SKIP_REBASE_P=1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+ \"@\n+ if (!$?) { exit(1) }\n+ displayName: 'Build & Test'\n+ env:\n+ HOME: $(Build.SourcesDirectory)\n+ MSYSTEM: MINGW64\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'Unmount test-cache'\n+ condition: true\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'windows'\n+ platform: Windows\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n - job: linux_clang\n displayName: linux-clang\n condition: succeeded()\n@@ -153,92 +239,6 @@ jobs:\n publishRunAttachments: false\n condition: succeededOrFailed()\n \n-- job: windows\n- displayName: Windows\n- condition: succeeded()\n- pool: Hosted\n- timeoutInMinutes: 240\n- steps:\n- - powershell: |\n- if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n- net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n- cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n- }\n- displayName: 'Mount test-cache'\n- env:\n- GITFILESHAREPWD: $(gitfileshare.pwd)\n- - powershell: |\n- # Helper to check the error level of the latest command (exit with error when appropriate)\n- function c() { if (!$?) { exit(1) } }\n-\n- # Add build agent's MinGit to PATH\n- $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n-\n- # Helper to initialize (or update) a Git worktree\n- function init ($path, $url, $set_origin) {\n- if (Test-Path $path) {\n- cd $path; c\n- if (Test-Path .git) {\n- & git init; c\n- } else {\n- & git status\n- }\n- } else {\n- & git init $path; c\n- cd $path; c\n- }\n- & git config core.autocrlf false; c\n- & git config core.untrackedCache true; c\n- if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n- & git remote add origin $url; c\n- }\n- & git fetch --depth=1 $url master; c\n- & git reset --hard FETCH_HEAD; c\n- & git clean -df; c\n- }\n-\n- # Initialize Git for Windows' SDK\n- $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n- init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n-\n- # Let Git ignore the SDK and the test-cache\n- \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n- displayName: 'Initialize the Git for Windows SDK'\n- - powershell: |\n- & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n- export MAKEFLAGS=-j10\n- export DEVELOPER=1\n- export NO_PERL=1\n- export NO_SVN_TESTS=1\n- export GIT_TEST_SKIP_REBASE_P=1\n-\n- ci/run-build-and-tests.sh || {\n- ci/print-test-failures.sh\n- exit 1\n- }\n- \"@\n- if (!$?) { exit(1) }\n- displayName: 'Build & Test'\n- env:\n- HOME: $(Build.SourcesDirectory)\n- MSYSTEM: MINGW64\n- - powershell: |\n- if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n- cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n- }\n- displayName: 'Unmount test-cache'\n- condition: true\n- env:\n- GITFILESHAREPWD: $(gitfileshare.pwd)\n- - task: PublishTestResults@2\n- displayName: 'Publish Test Results **/TEST-*.xml'\n- inputs:\n- mergeTestResults: true\n- testRunTitle: 'windows'\n- platform: Windows\n- publishRunAttachments: false\n- condition: succeededOrFailed()\n-\n - job: linux32\n displayName: Linux32\n condition: succeeded()\n-- \ngitgitgadget\n\n"},{"id":"366832","messageId":"ddee8ecd8a74de87339870f756d95913f086491c.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 09/21] Add a build definition for Azure DevOps","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:19Z","receivedAt":"2019-01-16T13:36:26Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis commit adds an azure-pipelines.yml file which is Azure DevOps'\nequivalent to Travis CI's .travis.yml.\n\nTo make things a bit easier to understand, we refrain from using the\n`matrix` feature here because (while it is powerful) it can be a bit\nconfusing to users who are not familiar with CI setups. Therefore, we\nuse a separate phase even for similar configurations (such as GCC vs\nClang on Linux, GCC vs Clang on macOS).\n\nAlso, we make use of the shiny new feature we just introduced where the\ntest suite can output JUnit-style .xml files. This information is made\navailable in a nice UI that allows the viewer to filter by phase and/or\ntest number, and to see trends such as: number of (failing) tests, time\nspent running the test suite, etc.\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 306 ++++++++++++++++++++++++++++++++++++++++++\n ci/lib.sh | 2 +-\n ci/mount-fileshare.sh | 25 ++++\n 3 files changed, 332 insertions(+), 1 deletion(-)\n create mode 100644 azure-pipelines.yml\n create mode 100755 ci/mount-fileshare.sh\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nnew file mode 100644\nindex 0000000000..f3cabb0dd0\n--- /dev/null\n+++ b/azure-pipelines.yml\n@@ -0,0 +1,306 @@\n+resources:\n+- repo: self\n+ fetchDepth: 1\n+\n+jobs:\n+- job: linux_clang\n+ displayName: linux-clang\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin &&\n+\n+ export CC=clang || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-clang'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: linux_gcc\n+ displayName: linux-gcc\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo add-apt-repository ppa:ubuntu-toolchain-r/test &&\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2 language-pack-is git-svn gcc-8 || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-gcc'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: osx_clang\n+ displayName: osx-clang\n+ condition: succeeded()\n+ pool: Hosted macOS\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ export CC=clang\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-clang'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: osx_gcc\n+ displayName: osx-gcc\n+ condition: succeeded()\n+ pool: Hosted macOS\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-gcc'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: gettext_poison\n+ displayName: GETTEXT_POISON\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev &&\n+\n+ export jobname=GETTEXT_POISON || exit 1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'gettext-poison'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: windows\n+ displayName: Windows\n+ condition: succeeded()\n+ pool: Hosted\n+ timeoutInMinutes: 240\n+ steps:\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n+ }\n+ displayName: 'Mount test-cache'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - powershell: |\n+ # Helper to check the error level of the latest command (exit with error when appropriate)\n+ function c() { if (!$?) { exit(1) } }\n+\n+ # Add build agent's MinGit to PATH\n+ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n+\n+ # Helper to initialize (or update) a Git worktree\n+ function init ($path, $url, $set_origin) {\n+ if (Test-Path $path) {\n+ cd $path; c\n+ if (Test-Path .git) {\n+ & git init; c\n+ } else {\n+ & git status\n+ }\n+ } else {\n+ & git init $path; c\n+ cd $path; c\n+ }\n+ & git config core.autocrlf false; c\n+ & git config core.untrackedCache true; c\n+ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n+ & git remote add origin $url; c\n+ }\n+ & git fetch --depth=1 $url master; c\n+ & git reset --hard FETCH_HEAD; c\n+ & git clean -df; c\n+ }\n+\n+ # Initialize Git for Windows' SDK\n+ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n+ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+\n+ # Let Git ignore the SDK and the test-cache\n+ \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n+ displayName: 'Initialize the Git for Windows SDK'\n+ - powershell: |\n+ & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ export MAKEFLAGS=-j10\n+ export DEVELOPER=1\n+ export NO_PERL=1\n+ export NO_SVN_TESTS=1\n+ export GIT_TEST_SKIP_REBASE_P=1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+ \"@\n+ if (!$?) { exit(1) }\n+ displayName: 'Build & Test'\n+ env:\n+ HOME: $(Build.SourcesDirectory)\n+ MSYSTEM: MINGW64\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'Unmount test-cache'\n+ condition: true\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'windows'\n+ platform: Windows\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: linux32\n+ displayName: Linux32\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo AGENT_OS=\"$AGENT_OS\" BUILD_BUILDNUMBER=\"$BUILD_BUILDNUMBER\" BUILD_REPOSITORY_URI=\"$BUILD_REPOSITORY_URI\" BUILD_SOURCEBRANCH=\"$BUILD_SOURCEBRANCH\" BUILD_SOURCEVERSION=\"$BUILD_SOURCEVERSION\" SYSTEM_PHASENAME=\"$SYSTEM_PHASENAME\" SYSTEM_TASKDEFINITIONSURI=\"$SYSTEM_TASKDEFINITIONSURI\" SYSTEM_TEAMPROJECT=\"$SYSTEM_TEAMPROJECT\" CC=$CC MAKEFLAGS=-j3 bash -lxc ci/run-linux32-docker.sh || exit 1\n+\n+ sudo chmod a+r t/out/TEST-*.xml\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-linux32-docker.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux32'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: static_analysis\n+ displayName: StaticAnalysis\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get install -y coccinelle &&\n+\n+ export jobname=StaticAnalysis &&\n+\n+ ci/run-static-analysis.sh || exit 1\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-static-analysis.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+\n+- job: documentation\n+ displayName: Documentation\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get install -y asciidoc xmlto asciidoctor &&\n+\n+ export ALREADY_HAVE_ASCIIDOCTOR=yes. &&\n+ export jobname=Documentation &&\n+\n+ ci/test-documentation.sh || exit 1\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/test-documentation.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex e560fae3aa..c5228c1547 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -42,7 +42,7 @@ then\n \n \tBREW_INSTALL_PACKAGES=\n \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n-\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n fi\n \n skip_branch_tip_with_tag () {\ndiff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\nnew file mode 100755\nindex 0000000000..26b58a8096\n--- /dev/null\n+++ b/ci/mount-fileshare.sh\n@@ -0,0 +1,25 @@\n+#!/bin/sh\n+\n+die () {\n+\techo \"$*\" >&2\n+\texit 1\n+}\n+\n+test $# = 4 ||\n+die \"Usage: $0 \"\n+\n+mkdir -p \"$4\" || die \"Could not create $4\"\n+\n+case \"$(uname -s)\" in\n+Linux)\n+\tsudo mount -t cifs -o vers=3.0,username=\"$2\",password=\"$3\",dir_mode=0777,file_mode=0777,serverino \"$1\" \"$4\"\n+\t;;\n+Darwin)\n+\tpass=\"$(echo \"$3\" | sed -e 's/\\//%2F/g' -e 's/+/%2B/g')\" &&\n+\tmount -t smbfs,soft \"smb://$2:$pass@${1#//}\" \"$4\"\n+\t;;\n+*)\n+\tdie \"No support for $(uname -s)\"\n+\t;;\n+esac ||\n+die \"Could not mount $4\"\n-- \ngitgitgadget\n\n"},{"id":"366834","messageId":"b6bb6a87a302ea146ce2fa0aba835781440dab12.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 11/21] ci: use git-sdk-64-minimal build artifact","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:20Z","receivedAt":"2019-01-16T13:36:28Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nInstead of a shallow fetch followed by a sparse checkout, we are\nbetter off by using a separate, dedicated Pipeline that bundles\nthe SDK as a build artifact, and then consuming that build artifact\nhere.\n\nIn fact, since this artifact will be used a lot, we spent substantial\ntime on figuring out a minimal subset of the Git for Windows SDK, just\nenough to build and test Git. The result is a size reduction from around\n1GB (compressed) to around 55MB (compressed). This also comes with the\nchange where we now call `usr\\bin\\bash.exe` directly, as `git-cmd.exe`\nis not included in the minimal SDK.\n\nThat reduces the time to initialize Git for Windows' SDK from anywhere\nbetween 2m30s-7m to a little over 1m.\n\nNote: in theory, we could also use the DownloadBuildArtifacts@0 task\nhere. However, restricted permissions that are in effect when building\nfrom forks would let this fail for PR builds, defeating the whole\npurpose of the Azure Pipelines support for git.git.\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 44 +++++++++-----------------------------------\n 1 file changed, 9 insertions(+), 35 deletions(-)\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex e44d2733a4..3085ff6ace 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -18,44 +18,18 @@ jobs:\n env:\n GITFILESHAREPWD: $(gitfileshare.pwd)\n - powershell: |\n- # Helper to check the error level of the latest command (exit with error when appropriate)\n- function c() { if (!$?) { exit(1) } }\n-\n- # Add build agent's MinGit to PATH\n- $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n-\n- # Helper to initialize (or update) a Git worktree\n- function init ($path, $url, $set_origin) {\n- if (Test-Path $path) {\n- cd $path; c\n- if (Test-Path .git) {\n- & git init; c\n- } else {\n- & git status\n- }\n- } else {\n- & git init $path; c\n- cd $path; c\n- }\n- & git config core.autocrlf false; c\n- & git config core.untrackedCache true; c\n- if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n- & git remote add origin $url; c\n- }\n- & git fetch --depth=1 $url master; c\n- & git reset --hard FETCH_HEAD; c\n- & git clean -df; c\n- }\n-\n- # Initialize Git for Windows' SDK\n- $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n- init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+ $urlbase = \"https://dev.azure.com/git-for-windows/git/_apis/build/builds\"\n+ $id = ((Invoke-WebRequest -UseBasicParsing \"${urlbase}?definitions=22&statusFilter=completed&resultFilter=succeeded&`$top=1\").content | ConvertFrom-JSON).value[0].id\n+ $downloadUrl = ((Invoke-WebRequest -UseBasicParsing \"${urlbase}/$id/artifacts\").content | ConvertFrom-JSON).value[1].resource.downloadUrl\n+ (New-Object Net.WebClient).DownloadFile($downloadUrl,\"git-sdk-64-minimal.zip\")\n+ Expand-Archive git-sdk-64-minimal.zip -DestinationPath . -Force\n+ Remove-Item git-sdk-64-minimal.zip\n \n # Let Git ignore the SDK and the test-cache\n- \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n- displayName: 'Initialize the Git for Windows SDK'\n+ \"/git-sdk-64-minimal/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n+ displayName: 'Download git-sdk-64-minimal'\n - powershell: |\n- & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n export MAKEFLAGS=-j10\n export DEVELOPER=1\n export NO_PERL=1\n-- \ngitgitgadget\n\n"},{"id":"366835","messageId":"df326039d9b76a1cdce7040d53e2faf4e6890c3a.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 13/21] README: add a build badge (status of the Azure Pipelines build)","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:23Z","receivedAt":"2019-01-16T13:36:29Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nJust like so many other OSS projects, we now also have a build badge.\n\nSigned-off-by: Johannes Schindelin \n---\n README.md | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/README.md b/README.md\nindex f920a42fad..764c480c66 100644\n--- a/README.md\n+++ b/README.md\n@@ -1,3 +1,5 @@\n+[![Build Status](https://dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2)\n+\n Git - fast, scalable, distributed revision control system\n =========================================================\n \n-- \ngitgitgadget\n\n"},{"id":"366836","messageId":"2b6ee78f021a47c5892d96b5d63d19bdedf25e2d.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 12/21] mingw: be more generous when wrapping up the setitimer() emulation","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:22Z","receivedAt":"2019-01-16T13:36:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nEvery once in a while, the Azure Pipeline fails with some semi-random\n\n\terror: timer thread did not terminate timely\n\nThis error message means that the thread that is used to emulate the\nsetitimer() function did not terminate within 1,000 milliseconds.\n\nThe most likely explanation (and therefore the one we should assume to\nbe true, according to Occam's Razor) is that the timeout of one second\nis simply not enough because we try to run so many tasks in parallel.\n\nSo let's give it ten seconds instead of only one. That should be enough.\n\nSigned-off-by: Johannes Schindelin \n---\n compat/mingw.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex b459e1a291..e0dfe8844d 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -2100,7 +2100,7 @@ static void stop_timer_thread(void)\n \tif (timer_event)\n \t\tSetEvent(timer_event);\t/* tell thread to terminate */\n \tif (timer_thread) {\n-\t\tint rc = WaitForSingleObject(timer_thread, 1000);\n+\t\tint rc = WaitForSingleObject(timer_thread, 10000);\n \t\tif (rc == WAIT_TIMEOUT)\n \t\t\terror(\"timer thread did not terminate timely\");\n \t\telse if (rc != WAIT_OBJECT_0)\n-- \ngitgitgadget\n\n"},{"id":"366837","messageId":"3a77eafb44117092a97121afabac5106bf58fcdd.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 19/21] tests: optionally skip bin-wrappers/","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:28Z","receivedAt":"2019-01-16T13:36:33Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis speeds up the tests by a bit on Windows, where running Unix shell\nscripts (and spawning processes) is not exactly a cheap operation.\n\nSigned-off-by: Johannes Schindelin \n---\n t/README | 9 +++++++++\n t/test-lib.sh | 19 +++++++++++++------\n 2 files changed, 22 insertions(+), 6 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 28711cc508..3f645b083a 100644\n--- a/t/README\n+++ b/t/README\n@@ -170,6 +170,15 @@ appropriately before running \"make\".\n \timplied by other options like --valgrind and\n \tGIT_TEST_INSTALLED.\n \n+--no-bin-wrappers::\n+\tBy default, the test suite uses the wrappers in\n+\t`../bin-wrappers/` to execute `git` and friends. With this option,\n+\t`../git` and friends are run directly. This is not recommended\n+\tin general, as the wrappers contain safeguards to ensure that no\n+\tfiles from an installed Git are used, but can speed up test runs\n+\tespecially on platforms where running shell scripts is expensive\n+\t(most notably, Windows).\n+\n --root=::\n \tCreate \"trash\" directories used to store all temporary data during\n \ttesting under , instead of the t/ directory.\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9113ad86b4..38a0f5d96f 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -294,6 +294,8 @@ do\n \t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t; shift ;;\n \t--with-dashes)\n \t\twith_dashes=t; shift ;;\n+\t--no-bin-wrappers)\n+\t\tno_bin_wrappers=t; shift ;;\n \t--no-color)\n \t\tcolor=; shift ;;\n \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n@@ -1080,16 +1082,21 @@ then\n \tPATH=$GIT_TEST_INSTALLED:$GIT_BUILD_DIR/t/helper:$PATH\n \tGIT_EXEC_PATH=${GIT_TEST_EXEC_PATH:-$GIT_EXEC_PATH}\n else # normal case, use ../bin-wrappers only unless $with_dashes:\n-\tgit_bin_dir=\"$GIT_BUILD_DIR/bin-wrappers\"\n-\tif ! test -x \"$git_bin_dir/git\"\n+\tif test -n \"$no_bin_wrappers\"\n \tthen\n-\t\tif test -z \"$with_dashes\"\n+\t\twith_dashes=t\n+\telse\n+\t\tgit_bin_dir=\"$GIT_BUILD_DIR/bin-wrappers\"\n+\t\tif ! test -x \"$git_bin_dir/git\"\n \t\tthen\n-\t\t\tsay \"$git_bin_dir/git is not executable; using GIT_EXEC_PATH\"\n+\t\t\tif test -z \"$with_dashes\"\n+\t\t\tthen\n+\t\t\t\tsay \"$git_bin_dir/git is not executable; using GIT_EXEC_PATH\"\n+\t\t\tfi\n+\t\t\twith_dashes=t\n \t\tfi\n-\t\twith_dashes=t\n+\t\tPATH=\"$git_bin_dir:$PATH\"\n \tfi\n-\tPATH=\"$git_bin_dir:$PATH\"\n \tGIT_EXEC_PATH=$GIT_BUILD_DIR\n \tif test -n \"$with_dashes\"\n \tthen\n-- \ngitgitgadget\n\n"},{"id":"366838","messageId":"b503167084e28b242a081d00f20ee2623bd1766a.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 17/21] tests: add t/helper/ to the PATH with --with-dashes","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:26Z","receivedAt":"2019-01-16T13:36:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWe really need to be able to find the test helpers... Really. This\nchange was forgotten when we moved the test helpers into t/helper/\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 531170011e..9113ad86b4 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1093,7 +1093,7 @@ else # normal case, use ../bin-wrappers only unless $with_dashes:\n \tGIT_EXEC_PATH=$GIT_BUILD_DIR\n \tif test -n \"$with_dashes\"\n \tthen\n-\t\tPATH=\"$GIT_BUILD_DIR:$PATH\"\n+\t\tPATH=\"$GIT_BUILD_DIR:$GIT_BUILD_DIR/t/helper:$PATH\"\n \tfi\n fi\n GIT_TEMPLATE_DIR=\"$GIT_BUILD_DIR\"/templates/blt\n-- \ngitgitgadget\n\n"},{"id":"366839","messageId":"fba97133bf476ce66eae77c962440efacee46d6d.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 20/21] ci: speed up Windows phase","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:28Z","receivedAt":"2019-01-16T13:36:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nAs Unix shell scripting comes at a hefty price on Windows, we have to\nsee where we can save some time to run the test suite.\n\nLet's skip the chain linting and the bin-wrappers/ redirection on\nWindows; this seems to shave of anywhere between 10-30% from the overall\nruntime.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex c5228c1547..bf5f1c9c0f 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -43,6 +43,8 @@ then\n \tBREW_INSTALL_PACKAGES=\n \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n+\ttest windows_nt != \"$CI_OS_NAME\" ||\n+\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n fi\n \n skip_branch_tip_with_tag () {\n-- \ngitgitgadget\n\n"},{"id":"366840","messageId":"e568349930b58f1da7e1aee84e137d2b55941b9c.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 21/21] ci: parallelize testing on Windows","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:30Z","receivedAt":"2019-01-16T13:36:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe fact that Git's test suite is implemented in Unix shell script that\nis as portable as we can muster, combined with the fact that Unix shell\nscripting is foreign to Windows (and therefore has to be emulated),\nresults in pretty abysmal speed of the test suite on that platform, for\npretty much no other reason than that language choice.\n\nFor comparison: while the Linux build & test is typically done within\nabout 8 minutes, the Windows build & test typically lasts about 80\nminutes in Azure Pipelines.\n\nTo help with that, let's use the Azure Pipeline feature where you can\nparallelize jobs, make jobs depend on each other, and pass artifacts\nbetween them.\n\nThe tests are distributed using the following heuristic: listing all\ntest scripts ordered by size in descending order (as a cheap way to\nestimate the overall run time), every Nth script is run (where N is the\ntotal number of parallel jobs), starting at the index corresponding to\nthe parallel job. This slicing is performed by a new function that is\nadded to the `test-tool`.\n\nTo optimize the overall runtime of the entire Pipeline, we need to move\nthe Windows jobs to the beginning (otherwise there would be a very\ndecent chance for the Pipeline to be run only the Windows build, while\nall the parallel Windows test jobs wait for this single one).\n\nWe use Azure Pipelines Artifacts for both the minimal Git for Windows\nSDK as well as the built executables, as deduplication and caching close\nto the agents makes that really fast. For comparison: while downloading\nand unpacking the minimal Git for Windows SDK via PowerShell takes only\none minute (down from anywhere between 2.5 to 7 when using a shallow\nclone), uploading it as Pipeline Artifact takes less than 30s and\ndownloading and unpacking less than 20s (sometimes even as little as\nonly twelve seconds).\n\nSigned-off-by: Johannes Schindelin \n---\n Makefile | 10 +++++\n azure-pipelines.yml | 82 +++++++++++++++++++++++++++++++++-----\n ci/make-test-artifacts.sh | 12 ++++++\n ci/run-test-slice.sh | 17 ++++++++\n t/helper/test-path-utils.c | 31 ++++++++++++++\n 5 files changed, 143 insertions(+), 9 deletions(-)\n create mode 100755 ci/make-test-artifacts.sh\n create mode 100755 ci/run-test-slice.sh\n\ndiff --git a/Makefile b/Makefile\nindex 044b4f77bd..daa318fe17 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2927,6 +2927,16 @@ rpm::\n \t@false\n .PHONY: rpm\n \n+artifacts-tar:: $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) \\\n+\t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n+\t\t$(NO_INSTALL) $(MOFILES)\n+\t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n+\t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n+\ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n+\tmkdir -p \"$(ARTIFACTS_DIRECTORY)\"\n+\t$(TAR) czf \"$(ARTIFACTS_DIRECTORY)/artifacts.tar.gz\" $^ templates/blt/\n+.PHONY: artifacts-tar\n+\n htmldocs = git-htmldocs-$(GIT_VERSION)\n manpages = git-manpages-$(GIT_VERSION)\n .PHONY: dist-doc distclean\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex 3085ff6ace..6cd27b3483 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -3,8 +3,8 @@ resources:\n fetchDepth: 1\n \n jobs:\n-- job: windows\n- displayName: Windows\n+- job: windows_build\n+ displayName: Windows Build\n condition: succeeded()\n pool: Hosted\n timeoutInMinutes: 240\n@@ -30,22 +30,86 @@ jobs:\n displayName: 'Download git-sdk-64-minimal'\n - powershell: |\n & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n- export MAKEFLAGS=-j10\n- export DEVELOPER=1\n- export NO_PERL=1\n- export NO_SVN_TESTS=1\n- export GIT_TEST_SKIP_REBASE_P=1\n+ ci/make-test-artifacts.sh artifacts\n+ \"@\n+ if (!$?) { exit(1) }\n+ displayName: Build\n+ env:\n+ HOME: $(Build.SourcesDirectory)\n+ MSYSTEM: MINGW64\n+ MAKEFLAGS: -j10\n+ DEVELOPER: 1\n+ NO_PERL: 1\n+ - task: PublishPipelineArtifact@0\n+ displayName: 'Publish Pipeline Artifact: test artifacts'\n+ inputs:\n+ artifactName: 'windows-artifacts'\n+ targetPath: '$(Build.SourcesDirectory)\\artifacts'\n+ - task: PublishPipelineArtifact@0\n+ displayName: 'Publish Pipeline Artifact: git-sdk-64-minimal'\n+ inputs:\n+ artifactName: 'git-sdk-64-minimal'\n+ targetPath: '$(Build.SourcesDirectory)\\git-sdk-64-minimal'\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'Unmount test-cache'\n+ condition: true\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+\n+- job: windows_test\n+ displayName: Windows Test\n+ dependsOn: windows_build\n+ condition: succeeded()\n+ pool: Hosted\n+ timeoutInMinutes: 240\n+ strategy:\n+ parallel: 10\n+ steps:\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n+ }\n+ displayName: 'Mount test-cache'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: DownloadPipelineArtifact@0\n+ displayName: 'Download Pipeline Artifact: test artifacts'\n+ inputs:\n+ artifactName: 'windows-artifacts'\n+ targetPath: '$(Build.SourcesDirectory)'\n+ - task: DownloadPipelineArtifact@0\n+ displayName: 'Download Pipeline Artifact: git-sdk-64-minimal'\n+ inputs:\n+ artifactName: 'git-sdk-64-minimal'\n+ targetPath: '$(Build.SourcesDirectory)\\git-sdk-64-minimal'\n+ - powershell: |\n+ & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n+ test -f artifacts.tar.gz || {\n+ echo No test artifacts found\\; skipping >&2\n+ exit 0\n+ }\n+ tar xf artifacts.tar.gz || exit 1\n+\n+ # Let Git ignore the SDK and the test-cache\n+ printf '%s\\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude\n \n- ci/run-build-and-tests.sh || {\n+ ci/run-test-slice.sh `$SYSTEM_JOBPOSITIONINPHASE `$SYSTEM_TOTALJOBSINPHASE || {\n ci/print-test-failures.sh\n exit 1\n }\n \"@\n if (!$?) { exit(1) }\n- displayName: 'Build & Test'\n+ displayName: 'Test (parallel)'\n env:\n HOME: $(Build.SourcesDirectory)\n MSYSTEM: MINGW64\n+ MAKEFLAGS: -j10\n+ NO_SVN_TESTS: 1\n+ GIT_TEST_SKIP_REBASE_P: 1\n - powershell: |\n if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\ndiff --git a/ci/make-test-artifacts.sh b/ci/make-test-artifacts.sh\nnew file mode 100755\nindex 0000000000..646967481f\n--- /dev/null\n+++ b/ci/make-test-artifacts.sh\n@@ -0,0 +1,12 @@\n+#!/bin/sh\n+#\n+# Build Git and store artifacts for testing\n+#\n+\n+mkdir -p \"$1\" # in case ci/lib.sh decides to quit early\n+\n+. ${0%/*}/lib.sh\n+\n+make artifacts-tar ARTIFACTS_DIRECTORY=\"$1\"\n+\n+check_unignored_build_artifacts\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nnew file mode 100755\nindex 0000000000..f8c2c3106a\n--- /dev/null\n+++ b/ci/run-test-slice.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+#\n+# Test Git in parallel\n+#\n+\n+. ${0%/*}/lib.sh\n+\n+case \"$CI_OS_NAME\" in\n+windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n+*) ln -s \"$cache_dir/.prove\" t/.prove;;\n+esac\n+\n+make --quiet -C t T=\"$(cd t &&\n+\t./helper/test-tool path-utils slice-tests \"$1\" \"$2\" t[0-9]*.sh |\n+\ttr '\\n' ' ')\"\n+\n+check_unignored_build_artifacts\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex 30211d6d64..f4c5775090 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -177,6 +177,14 @@ static int is_dotgitmodules(const char *path)\n \treturn is_hfs_dotgitmodules(path) || is_ntfs_dotgitmodules(path);\n }\n \n+static int cmp_by_st_size(const void *a, const void *b)\n+{\n+\tintptr_t x = (intptr_t)((struct string_list_item *)a)->util;\n+\tintptr_t y = (intptr_t)((struct string_list_item *)b)->util;\n+\n+\treturn x > y ? -1 : (x < y ? +1 : 0);\n+}\n+\n int cmd__path_utils(int argc, const char **argv)\n {\n \tif (argc == 3 && !strcmp(argv[1], \"normalize_path_copy\")) {\n@@ -303,6 +311,29 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn !!res;\n \t}\n \n+\tif (argc > 5 && !strcmp(argv[1], \"slice-tests\")) {\n+\t\tint res = 0;\n+\t\tlong offset, stride, i;\n+\t\tstruct string_list list = STRING_LIST_INIT_NODUP;\n+\t\tstruct stat st;\n+\n+\t\toffset = strtol(argv[2], NULL, 10);\n+\t\tstride = strtol(argv[3], NULL, 10);\n+\t\tif (stride < 1)\n+\t\t\tstride = 1;\n+\t\tfor (i = 4; i < argc; i++)\n+\t\t\tif (stat(argv[i], &st))\n+\t\t\t\tres = error_errno(\"Cannot stat '%s'\", argv[i]);\n+\t\t\telse\n+\t\t\t\tstring_list_append(&list, argv[i])->util =\n+\t\t\t\t\t(void *)(intptr_t)st.st_size;\n+\t\tQSORT(list.items, list.nr, cmp_by_st_size);\n+\t\tfor (i = offset; i < list.nr; i+= stride)\n+\t\t\tprintf(\"%s\\n\", list.items[i].string);\n+\n+\t\treturn !!res;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\n-- \ngitgitgadget\n"},{"id":"366841","messageId":"713910e1dc634e3c09b374759a3bd282946eca96.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 18/21] t0061: fix with --with-dashes and RUNTIME_PREFIX","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:27Z","receivedAt":"2019-01-16T13:36:39Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWhen building Git with RUNTIME_PREFIX and starting a test helper from\nt/helper/, it fails to detect the system prefix correctly.\n\nThis is the reason that the warning\n\n\tRUNTIME_PREFIX requested, but prefix computation failed. [...]\n\nto be printed.\n\nIn t0061, we did not expect that to happen, and it actually did not\nhappen in the normal case, because bin-wrappers/test-tool specifically\nsets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool wants\nto know about the runtime prefix).\n\nHowever, with --with-dashes, bin-wrappers/test-tool is no longer called,\nbut t/helper/test-tool is called directly.\n\nSo let's just ignore the RUNTIME_PREFIX warning.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t0061-run-command.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 99a614bc7c..5a2d087bf0 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -166,7 +166,8 @@ test_trace () {\n \texpect=\"$1\"\n \tshift\n \tGIT_TRACE=1 test-tool run-command \"$@\" run-command true 2>&1 >/dev/null | \\\n-\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' >actual &&\n+\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' \\\n+\t\t\t-e '/RUNTIME_PREFIX requested/d' >actual &&\n \techo \"$expect true\" >expect &&\n \ttest_cmp expect actual\n }\n-- \ngitgitgadget\n\n"},{"id":"366842","messageId":"272c0c044617977c838dcf97d60f9315f8acb36f.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 07/21] tests: optionally write results as JUnit-style .xml","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:17Z","receivedAt":"2019-01-16T13:36:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis will come in handy when publishing the results of Git's test suite\nduring an automated Azure DevOps run.\n\nNote: we need to make extra sure that invalid UTF-8 encoding is turned\ninto valid UTF-8 (using the Replacement Character, \\uFFFD) because\nt9902's trace contains such invalid byte sequences, and the task in the\nAzure Pipeline that uploads the test results would refuse to do anything\nif it was asked to parse an .xml file with invalid UTF-8 in it.\n\nSigned-off-by: Johannes Schindelin \n---\n Makefile | 1 +\n t/.gitignore | 1 +\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/helper/test-xml-encode.c | 80 +++++++++++++++++++++++++++++++++\n t/test-lib.sh | 91 ++++++++++++++++++++++++++++++++++++++\n 6 files changed, 175 insertions(+)\n create mode 100644 t/helper/test-xml-encode.c\n\ndiff --git a/Makefile b/Makefile\nindex 1a44c811aa..044b4f77bd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -754,6 +754,7 @@ TEST_BUILTINS_OBJS += test-submodule-config.o\n TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o\n TEST_BUILTINS_OBJS += test-subprocess.o\n TEST_BUILTINS_OBJS += test-urlmatch-normalization.o\n+TEST_BUILTINS_OBJS += test-xml-encode.o\n TEST_BUILTINS_OBJS += test-wildmatch.o\n TEST_BUILTINS_OBJS += test-windows-named-pipe.o\n TEST_BUILTINS_OBJS += test-write-cache.o\ndiff --git a/t/.gitignore b/t/.gitignore\nindex 348715f0e4..91cf5772fe 100644\n--- a/t/.gitignore\n+++ b/t/.gitignore\n@@ -2,3 +2,4 @@\n /test-results\n /.prove\n /chainlinttmp\n+/out/\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex bfb195b1a8..4b4b397d93 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -49,6 +49,7 @@ static struct test_cmd cmds[] = {\n \t{ \"submodule-nested-repo-config\", cmd__submodule_nested_repo_config },\n \t{ \"subprocess\", cmd__subprocess },\n \t{ \"urlmatch-normalization\", cmd__urlmatch_normalization },\n+\t{ \"xml-encode\", cmd__xml_encode },\n \t{ \"wildmatch\", cmd__wildmatch },\n #ifdef GIT_WINDOWS_NATIVE\n \t{ \"windows-named-pipe\", cmd__windows_named_pipe },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 042f12464b..c0ab65e370 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -45,6 +45,7 @@ int cmd__submodule_config(int argc, const char **argv);\n int cmd__submodule_nested_repo_config(int argc, const char **argv);\n int cmd__subprocess(int argc, const char **argv);\n int cmd__urlmatch_normalization(int argc, const char **argv);\n+int cmd__xml_encode(int argc, const char **argv);\n int cmd__wildmatch(int argc, const char **argv);\n #ifdef GIT_WINDOWS_NATIVE\n int cmd__windows_named_pipe(int argc, const char **argv);\ndiff --git a/t/helper/test-xml-encode.c b/t/helper/test-xml-encode.c\nnew file mode 100644\nindex 0000000000..367c4875e6\n--- /dev/null\n+++ b/t/helper/test-xml-encode.c\n@@ -0,0 +1,80 @@\n+#include \"test-tool.h\"\n+\n+static const char *utf8_replace_character = \"�\";\n+\n+/*\n+ * Encodes (possibly incorrect) UTF-8 on to , to be embedded\n+ * in an XML file.\n+ */\n+int cmd__xml_encode(int argc, const char **argv)\n+{\n+\tunsigned char buf[1024], tmp[4], *tmp2 = NULL;\n+\tssize_t cur = 0, len = 1, remaining = 0;\n+\tunsigned char ch;\n+\n+\tfor (;;) {\n+\t\tif (++cur == len) {\n+\t\t\tlen = xread(0, buf, sizeof(buf));\n+\t\t\tif (!len)\n+\t\t\t\treturn 0;\n+\t\t\tif (len < 0)\n+\t\t\t\tdie_errno(\"Could not read \");\n+\t\t\tcur = 0;\n+\t\t}\n+\t\tch = buf[cur];\n+\n+\t\tif (tmp2) {\n+\t\t\tif ((ch & 0xc0) != 0x80) {\n+\t\t\t\tfputs(utf8_replace_character, stdout);\n+\t\t\t\ttmp2 = 0;\n+\t\t\t\tcur--;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\t*tmp2 = ch;\n+\t\t\ttmp2++;\n+\t\t\tif (--remaining == 0) {\n+\t\t\t\tfwrite(tmp, tmp2 - tmp, 1, stdout);\n+\t\t\t\ttmp2 = 0;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!(ch & 0x80)) {\n+\t\t\t/* 0xxxxxxx */\n+\t\t\tif (ch == '&')\n+\t\t\t\tfputs(\"&\", stdout);\n+\t\t\telse if (ch == '\\'')\n+\t\t\t\tfputs(\"'\", stdout);\n+\t\t\telse if (ch == '\"')\n+\t\t\t\tfputs(\""\", stdout);\n+\t\t\telse if (ch == '<')\n+\t\t\t\tfputs(\"<\", stdout);\n+\t\t\telse if (ch == '>')\n+\t\t\t\tfputs(\">\", stdout);\n+\t\t\telse if (ch >= 0x20)\n+\t\t\t\tfputc(ch, stdout);\n+\t\t\telse if (ch == 0x09 || ch == 0x0a || ch == 0x0d)\n+\t\t\t\tfprintf(stdout, \"&#x%02x;\", ch);\n+\t\t\telse\n+\t\t\t\tfputs(utf8_replace_character, stdout);\n+\t\t} else if ((ch & 0xe0) == 0xc0) {\n+\t\t\t/* 110XXXXx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 1;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else if ((ch & 0xf0) == 0xe0) {\n+\t\t\t/* 1110XXXX 10Xxxxxx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 2;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else if ((ch & 0xf8) == 0xf0) {\n+\t\t\t/* 11110XXX 10XXxxxx 10xxxxxx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 3;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else\n+\t\t\tfputs(utf8_replace_character, stdout);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 0f1faa24b2..e9782b6b32 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -339,6 +339,9 @@ do\n \t-V|--verbose-log)\n \t\tverbose_log=t\n \t\tshift ;;\n+\t--write-junit-xml)\n+\t\twrite_junit_xml=t\n+\t\tshift ;;\n \t*)\n \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n \tesac\n@@ -486,11 +489,24 @@ trap 'exit $?' INT\n # the test_expect_* functions instead.\n \n test_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$*\"\n+\tfi\n \ttest_success=$(($test_success + 1))\n \tsay_color \"\" \"ok $test_count - $@\"\n }\n \n test_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_insert=\"\"\n+\t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n+\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\tjunit_insert=\"$junit_insert\"\n+\t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n+\tfi\n \ttest_failure=$(($test_failure + 1))\n \tsay_color error \"not ok $test_count - $1\"\n \tshift\n@@ -499,11 +515,19 @@ test_failure_ () {\n }\n \n test_known_broken_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (breakage fixed)\"\n+\tfi\n \ttest_fixed=$(($test_fixed+1))\n \tsay_color error \"ok $test_count - $@ # TODO known breakage vanished\"\n }\n \n test_known_broken_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (known breakage)\"\n+\tfi\n \ttest_broken=$(($test_broken+1))\n \tsay_color warn \"not ok $test_count - $@ # TODO known breakage\"\n }\n@@ -761,6 +785,10 @@ test_start_ () {\n \ttest_count=$(($test_count+1))\n \tmaybe_setup_verbose\n \tmaybe_setup_valgrind\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_start=$(test-tool date getnanos)\n+\tfi\n }\n \n test_finish_ () {\n@@ -798,6 +826,13 @@ test_skip () {\n \n \tcase \"$to_skip\" in\n \tt)\n+\t\tif test -n \"$write_junit_xml\"\n+\t\tthen\n+\t\t\tmessage=\"$(xml_attr_encode \"$skipped_reason\")\"\n+\t\t\twrite_junit_xml_testcase \"$1\" \\\n+\t\t\t\t\" \"\n+\t\tfi\n+\n \t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n@@ -813,9 +848,51 @@ test_at_end_hook_ () {\n \t:\n }\n \n+write_junit_xml () {\n+\tcase \"$1\" in\n+\t--truncate)\n+\t\t>\"$junit_xml_path\"\n+\t\tjunit_have_testcase=\n+\t\tshift\n+\t\t;;\n+\tesac\n+\tprintf '%s\\n' \"$@\" >>\"$junit_xml_path\"\n+}\n+\n+xml_attr_encode () {\n+\tprintf '%s\\n' \"$@\" | test-tool xml-encode\n+}\n+\n+write_junit_xml_testcase () {\n+\tjunit_attrs=\"name=\\\"$(xml_attr_encode \"$this_test.$test_count $1\")\\\"\"\n+\tshift\n+\tjunit_attrs=\"$junit_attrs classname=\\\"$this_test\\\"\"\n+\tjunit_attrs=\"$junit_attrs time=\\\"$(test-tool \\\n+\t\tdate getnanos $junit_start)\\\"\"\n+\twrite_junit_xml \"$(printf '%s\\n' \\\n+\t\t\" \" \"$@\" \" \")\"\n+\tjunit_have_testcase=t\n+}\n+\n test_done () {\n \tGIT_EXIT_OK=t\n \n+\tif test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n+\tthen\n+\t\ttest -n \"$junit_have_testcase\" || {\n+\t\t\tjunit_start=$(test-tool date getnanos)\n+\t\t\twrite_junit_xml_testcase \"all tests skipped\"\n+\t\t}\n+\n+\t\t# adjust the overall time\n+\t\tjunit_time=$(test-tool date getnanos $junit_suite_start)\n+\t\tsed \"s/]*/& time=\\\"$junit_time\\\"/\" \\\n+\t\t\t<\"$junit_xml_path\" >\"$junit_xml_path.new\"\n+\t\tmv \"$junit_xml_path.new\" \"$junit_xml_path\"\n+\n+\t\twrite_junit_xml \" \" \"\"\n+\tfi\n+\n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n \t\ttest_results_dir=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n@@ -1051,6 +1128,7 @@ then\n else\n \tmkdir -p \"$TRASH_DIRECTORY\"\n fi\n+\n # Use -P to resolve symlinks in our working directory so that the cwd\n # in subprocesses like git equals our $PWD (for pathname comparisons).\n cd -P \"$TRASH_DIRECTORY\" || exit 1\n@@ -1064,6 +1142,19 @@ then\n \ttest_done\n fi\n \n+if test -n \"$write_junit_xml\"\n+then\n+\tjunit_xml_dir=\"$TEST_OUTPUT_DIRECTORY/out\"\n+\tmkdir -p \"$junit_xml_dir\"\n+\tjunit_xml_base=${0##*/}\n+\tjunit_xml_path=\"$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml\"\n+\tjunit_attrs=\"name=\\\"${junit_xml_base%.sh}\\\"\"\n+\tjunit_attrs=\"$junit_attrs timestamp=\\\"$(TZ=UTC \\\n+\t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n+\twrite_junit_xml --truncate \"\" \" \"\n+\tjunit_suite_start=$(test-tool date getnanos)\n+fi\n+\n # Provide an implementation of the 'yes' utility\n yes () {\n \tif test $# = 0\n-- \ngitgitgadget\n\n"},{"id":"366843","messageId":"0d547db8f75fa38223712355b6f7152acd3254a4.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 14/21] tests: avoid calling Perl just to determine file sizes","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:23Z","receivedAt":"2019-01-16T13:36:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIt is a bit ridiculous to spin up a full-blown Perl instance (especially\non Windows, where that means spinning up a full POSIX emulation layer,\nAKA the MSYS2 runtime) just to tell how large a given file is.\n\nSo let's just use the test-tool to do that job instead.\n\nThis command will also be used over the next commits, to allow for\ncutting out individual test cases' verbose log from the file generated\nvia --verbose-log.\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-path-utils.c | 12 ++++++++++++\n t/t0021-conversion.sh | 2 +-\n t/t1050-large.sh | 2 +-\n t/t5315-pack-objects-compression.sh | 2 +-\n t/t9303-fast-import-compression.sh | 2 +-\n 5 files changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex ae091d9b3e..30211d6d64 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -291,6 +291,18 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn !!res;\n \t}\n \n+\tif (argc > 2 && !strcmp(argv[1], \"file-size\")) {\n+\t\tint res = 0, i;\n+\t\tstruct stat st;\n+\n+\t\tfor (i = 2; i < argc; i++)\n+\t\t\tif (stat(argv[i], &st))\n+\t\t\t\tres = error_errno(\"Cannot stat '%s'\", argv[i]);\n+\t\t\telse\n+\t\t\t\tprintf(\"%\"PRIuMAX\"\\n\", (uintmax_t)st.st_size);\n+\t\treturn !!res;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex fd5f1ac649..e10f5f787f 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -24,7 +24,7 @@ generate_random_characters () {\n }\n \n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n filter_git () {\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 1a9b21b293..dcb4dbba67 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -8,7 +8,7 @@ test_description='adding and checking out large blobs'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n test_expect_success setup '\ndiff --git a/t/t5315-pack-objects-compression.sh b/t/t5315-pack-objects-compression.sh\nindex 34c47dae09..df970d7584 100755\n--- a/t/t5315-pack-objects-compression.sh\n+++ b/t/t5315-pack-objects-compression.sh\n@@ -7,7 +7,7 @@ test_description='pack-object compression configuration'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n test_expect_success setup '\ndiff --git a/t/t9303-fast-import-compression.sh b/t/t9303-fast-import-compression.sh\nindex 856219f46a..5045f02a53 100755\n--- a/t/t9303-fast-import-compression.sh\n+++ b/t/t9303-fast-import-compression.sh\n@@ -6,7 +6,7 @@ test_description='compression setting of fast-import utility'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n import_large () {\n-- \ngitgitgadget\n\n"},{"id":"366844","messageId":"f678b105f81ec60f64c0f19e3cc7a8ef151ed52e.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 15/21] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:24Z","receivedAt":"2019-01-16T13:36:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe JUnit XML format lends itself to be presented in a powerful UI,\nwhere you can drill down to the information you are interested in very\nquickly.\n\nFor test failures, this usually means that you want to see the detailed\ntrace of the failing tests.\n\nWith Travis CI, we passed the `--verbose-log` option to get those\ntraces. However, that seems excessive, as we do not need/use the logs in\nalmost all of those cases: only when a test fails do we have a way to\ninclude the trace.\n\nSo let's do something different when using Azure DevOps: let's run all\nthe tests with `--quiet` first, and only if a failure is encountered,\ntry to trace the commands as they are executed.\n\nOf course, we cannot turn on `--verbose-log` after the fact. So let's\njust re-run the test with all the same options, adding `--verbose-log`.\nAnd then munging the output file into the JUnit XML on the fly.\n\nNote: there is an off chance that re-running the test in verbose mode\n\"fixes\" the failures (and this does happen from time to time!). That is\na possibility we should be able to live with. Ideally, we would label\nthis as \"Passed upon rerun\", and Azure Pipelines even know about that\noutcome, but it is not available when using the JUnit XML format for\nnow:\nhttps://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 22 +++++++++++++++++++++-\n 1 file changed, 21 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex e9782b6b32..f5371f505a 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -503,8 +503,18 @@ test_failure_ () {\n \t\tjunit_insert=\"\"\n \t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n-\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\t\t\"$(if test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\t\t then\n+\t\t\t\tcut -c \"$GIT_TEST_TEE_OFFSET-\" <\"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\t\t else\n+\t\t\t\tprintf '%s\\n' \"$@\" | sed 1d\n+\t\t\t fi)\")\"\n \t\tjunit_insert=\"$junit_insert\"\n+\t\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\tthen\n+\t\t\tjunit_insert=\"$junit_insert$(xml_attr_encode \\\n+\t\t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n+\t\tfi\n \t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n \tfi\n \ttest_failure=$(($test_failure + 1))\n@@ -872,6 +882,11 @@ write_junit_xml_testcase () {\n \twrite_junit_xml \"$(printf '%s\\n' \\\n \t\t\" \" \"$@\" \" \")\"\n \tjunit_have_testcase=t\n+\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\tthen\n+\t\tGIT_TEST_TEE_OFFSET=$(test-tool path-utils file-size \\\n+\t\t\t\"$GIT_TEST_TEE_OUTPUT_FILE\")\n+\tfi\n }\n \n test_done () {\n@@ -1153,6 +1168,11 @@ then\n \t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n \twrite_junit_xml --truncate \"\" \" \"\n \tjunit_suite_start=$(test-tool date getnanos)\n+\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\tthen\n+\t\tGIT_TEST_TEE_OFFSET=0\n+\t\tGIT_TEST_TEE_ERR_OFFSET=0\n+\tfi\n fi\n \n # Provide an implementation of the 'yes' utility\n-- \ngitgitgadget\n\n"},{"id":"366845","messageId":"8ef674a2364a624eeb0b6f279a088d4c4bc78d44.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 16/21] mingw: try to work around issues with the test cleanup","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:25Z","receivedAt":"2019-01-16T13:36:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIt seems that every once in a while in the Git for Windows SDK, there\nare some transient file locking issues preventing the test clean up to\ndelete the trash directory. Let's be gentle and try again five seconds\nlater, and only error out if it still fails the second time.\n\nThis change helps Windows, and does not hurt any other platform\n(normally, it is highly unlikely that said deletion fails, and if it\ndoes, normally it will fail again even 5 seconds later).\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex f5371f505a..531170011e 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -970,7 +970,11 @@ test_done () {\n \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n \n \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n-\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n+\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n+\t\t\t\t# try again in a bit\n+\t\t\t\tsleep 5;\n+\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n+\t\t\t} ||\n \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n \t\tfi\n \t\ttest_at_end_hook_\n-- \ngitgitgadget\n\n"},{"id":"366846","messageId":"fefab79d4634f302e1b3b5f3e2c48e689f148265.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 08/21] ci/lib.sh: add support for Azure Pipelines","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:18Z","receivedAt":"2019-01-16T13:36:46Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis patch introduces a conditional arm that defines some environment\nvariables and a function that displays the URL given the job id (to\nidentify previous runs for known-good trees).\n\nFor example, we do not have to install the git-lfs and gettext packages\non Azure Pipelines' macOS agents: they are already installed, and trying\nto install them again would result in an error.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex bfc406a63b..e560fae3aa 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -20,6 +20,29 @@ then\n \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+elif test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n+then\n+\t# We are running in Azure Pipelines\n+\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n+\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n+\tCI_JOB_ID=\"$BUILD_BUILDID\"\n+\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n+\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n+\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n+\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n+\tCC=\"${CC:-gcc}\"\n+\n+\t# use a subdirectory of the cache dir (because the file share is shared\n+\t# among *all* phases)\n+\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n+\n+\turl_for_job_id () {\n+\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n fi\n \n skip_branch_tip_with_tag () {\n-- \ngitgitgadget\n\n"},{"id":"366847","messageId":"a35bc43a04b0251f6f5a3976e062b606b50b6009.1547645770.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v3 05/21] ci: use a junction on Windows instead of a symlink","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-16T13:36:15Z","receivedAt":"2019-01-16T13:36:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nSymbolic links are still not quite as easy to use on Windows as on Linux\n(for example, on versions older than Windows 10, only administrators can\ncreate symlinks, and on Windows 10 you still need to be in developer\nmode for regular users to have permission), but NTFS junctions can give\nus a way out.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/run-build-and-tests.sh | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 80d72d120f..74d838ea01 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -5,7 +5,10 @@\n \n . ${0%/*}/lib.sh\n \n-ln -s \"$cache_dir/.prove\" t/.prove\n+case \"$CI_OS_NAME\" in\n+windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n+*) ln -s \"$cache_dir/.prove\" t/.prove;;\n+esac\n \n make\n make --quiet test\n-- \ngitgitgadget\n\n"},{"id":"366851","messageId":"nycvar.QRO.7.76.6.1901161503121.41@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqo9biigod.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-16T14:04:05Z","receivedAt":"2019-01-16T14:04:27Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Thu, 25 Oct 2018, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > For a long time already, we have Git's source code continuously tested via\n> > Travis CI, see e.g. https://travis-ci.org/git/git/builds/421738884. It has\n> > served us well, and more and more developers actually pay attention and\n> > benefit from the testing this gives us.\n> \n> What's the current status of this topic?\n> [...]\n> \n> The topic was marked as \"On hold, monitoring discussion\" and it seems\n> that discussion has quieted down, so the next step is to see an updated\n> series?\n\nSee the updated series:\nhttps://public-inbox.org/git/pull.31.v3.git.gitgitgadget@gmail.com/\n\nThanks,\nDscho\n"},{"id":"366907","messageId":"nycvar.QRO.7.76.6.1901171112000.41@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"f678b105f81ec60f64c0f19e3cc7a8ef151ed52e.1547645770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 15/21] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-17T10:18:15Z","receivedAt":"2019-01-17T10:18:39Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 16 Jan 2019, Johannes Schindelin via GitGitGadget wrote:\n\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index e9782b6b32..f5371f505a 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -503,8 +503,18 @@ test_failure_ () {\n> \t\tjunit_insert=\" \t\tjunit_insert=\"$junit_insert $(xml_attr_encode \"$1\")\\\">\"\n> \t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n> -\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n> +\t\t\t\"$(if test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n> +\t\t\t then\n> +\t\t\t\tcut -c \"$GIT_TEST_TEE_OFFSET-\" <\"$GIT_TEST_TEE_OUTPUT_FILE\"\n\nOoops. This does not work.\n\nThe intention of this patch was to remember the file size of the tee\noutput file at the point when the previous test case was done, and skip\nthat part (so that the first thing the user sees is the detailed trace of\n*just* the failed test case, with the complete trace being attached as\n`Standard_Error_Output.log`).\n\nHowever, I totally misunderstood that `cut` is *always* line based. So\nthis `cut -c` (which should have been a `cut -b` to begin with) is just\nplain wrong.\n\nAnd since there is no easy Unix shell tool to just skip a bunch of bytes\n(`dd` comes close, but you have to abuse the buffer size to specify the\nnumber of bytes, or use buffer size 1, neither of which I like), I will\njust add a dozen lines to the `test-path-utils.c` to get problem fixed.\n\n> +\t\t\t else\n> +\t\t\t\tprintf '%s\\n' \"$@\" | sed 1d\n> +\t\t\t fi)\")\"\n> \t\tjunit_insert=\"$junit_insert\"\n> +\t\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n> +\t\tthen\n> +\t\t\tjunit_insert=\"$junit_insert$(xml_attr_encode \\\n> +\t\t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n> +\t\tfi\n> \t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n> \tfi\n> \ttest_failure=$(($test_failure + 1))\n> @@ -872,6 +882,11 @@ write_junit_xml_testcase () {\n> \twrite_junit_xml \"$(printf '%s\\n' \\\n> \t\t\" \" \"$@\" \" \")\"\n> \tjunit_have_testcase=t\n> +\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n> +\tthen\n> +\t\tGIT_TEST_TEE_OFFSET=$(test-tool path-utils file-size \\\n> +\t\t\t\"$GIT_TEST_TEE_OUTPUT_FILE\")\n> +\tfi\n> }\n> \n> test_done () {\n> @@ -1153,6 +1168,11 @@ then\n> \t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n> \twrite_junit_xml --truncate \"\" \" \"\n> \tjunit_suite_start=$(test-tool date getnanos)\n> +\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n> +\tthen\n> +\t\tGIT_TEST_TEE_OFFSET=0\n> +\t\tGIT_TEST_TEE_ERR_OFFSET=0\n\nAlso, in case anybody wonders, this `GIT_TEST_TEE_ERR_OFFSET` variable is\nno longer used, and I should have removed it.\n\nOh well, I will just queue up the fixes in preparation for v4.\n\nCiao,\nDscho\n\n> +\tfi\n> fi\n> \n> # Provide an implementation of the 'yes' utility\n> -- \n> gitgitgadget\n> \n> \n"},{"id":"367027","messageId":"xmqqsgxrm2c0.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"75ec97b3921f3ed346e9ab119ebff2546f03fade.1547645770.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 01/21] travis: fix skipping tagged releases","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-17T20:55:11Z","receivedAt":"2019-01-17T20:55:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> When building a PR, TRAVIS_BRANCH refers to the *target branch*.\n> Therefore, if a PR targets `master`, and `master` happened to be tagged,\n> we skipped the build by mistake.\n\nGood spotting.\n\n>\n> Fix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*)\n> when available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also\n> known as \"push builds\").\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> ci/lib-travisci.sh | 4 ++++\n> 1 file changed, 4 insertions(+)\n>\n> diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh\n> index 69dff4d1ec..d9d4f1a9d7 100755\n> --- a/ci/lib-travisci.sh\n> +++ b/ci/lib-travisci.sh\n> @@ -1,5 +1,9 @@\n> # Library of functions shared by all CI scripts\n> \n> +# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n> +# want here. We want the source branch instead.\n> +TRAVIS_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n\nTo those who are familiar with Travis [*1*], I suspect that\nTRAVIS_BRANCH has a specific meaning (e.g. with PR, it is target\nbranch), and assigning a different value that is taken from another\nvariable with different meanings to TravisCI would confuse them.\n\nPerhaps introduce a new variable, like so...\n\n\tBRANCH_TO_BUILD=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n\nand then s/TRAVIS_BRANCH/BRANCH_TO_BUILD/g on the body of the\nskip_branch_tip_with_tag helper function.\n\nThanks.\n\n\n[Footnote]\n\n*1* I obviously am not among them; otherwise we would have caught\nthis while reviewing 09f5e974 (\"travis-ci: skip a branch build if\nequal tag is present\", 2017-09-10).\n\n\n\n\n> skip_branch_tip_with_tag () {\n> \t# Sometimes, a branch is pushed at the same time the tag that points\n> \t# at the same commit as the tip of the branch is pushed, and building\n"},{"id":"367031","messageId":"xmqqk1j3m25f.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1901161503121.41@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-17T20:59:08Z","receivedAt":"2019-01-17T20:59:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> ...\n> See the updated series:\n> https://public-inbox.org/git/pull.31.v3.git.gitgitgadget@gmail.com/\n\nThanks.\n\nI see that you are already planning for v4, but I'll find time to\ntake a look at what is posted sometime this week anyway.\n"},{"id":"367078","messageId":"nycvar.QRO.7.76.6.1901181104070.41@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqsgxrm2c0.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 01/21] travis: fix skipping tagged releases","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-18T10:05:17Z","receivedAt":"2019-01-18T10:05:39Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Thu, 17 Jan 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > Fix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*)\n> > when available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also\n> > known as \"push builds\").\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > ci/lib-travisci.sh | 4 ++++\n> > 1 file changed, 4 insertions(+)\n> >\n> > diff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh\n> > index 69dff4d1ec..d9d4f1a9d7 100755\n> > --- a/ci/lib-travisci.sh\n> > +++ b/ci/lib-travisci.sh\n> > @@ -1,5 +1,9 @@\n> > # Library of functions shared by all CI scripts\n> > \n> > +# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n> > +# want here. We want the source branch instead.\n> > +TRAVIS_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> \n> To those who are familiar with Travis [*1*], I suspect that\n> TRAVIS_BRANCH has a specific meaning (e.g. with PR, it is target\n> branch), and assigning a different value that is taken from another\n> variable with different meanings to TravisCI would confuse them.\n> \n> Perhaps introduce a new variable, like so...\n> \n> \tBRANCH_TO_BUILD=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> \n> and then s/TRAVIS_BRANCH/BRANCH_TO_BUILD/g on the body of the\n> skip_branch_tip_with_tag helper function.\n\nI already introduce such a variable later in the same patch series:\n`CI_BRANCH`. It is a good idea to introduce that already here, as it\ndeclutters that later patch a bit, I would think.\n\nWill change,\nDscho\n\n> \n> Thanks.\n> \n> \n> [Footnote]\n> \n> *1* I obviously am not among them; otherwise we would have caught\n> this while reviewing 09f5e974 (\"travis-ci: skip a branch build if\n> equal tag is present\", 2017-09-10).\n> \n> \n> \n> \n> > skip_branch_tip_with_tag () {\n> > \t# Sometimes, a branch is pushed at the same time the tag that points\n> > \t# at the same commit as the tip of the branch is pushed, and building\n> \n"},{"id":"367084","messageId":"nycvar.QRO.7.76.6.1901181246130.41@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqk1j3m25f.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 00/13] Offer to run CI/PR builds in Azure Pipelines","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-18T11:49:10Z","receivedAt":"2019-01-18T11:49:30Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Thu, 17 Jan 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n> \n> > ...\n> > See the updated series:\n> > https://public-inbox.org/git/pull.31.v3.git.gitgitgadget@gmail.com/\n> \n> Thanks.\n> \n> I see that you are already planning for v4, but I'll find time to\n> take a look at what is posted sometime this week anyway.\n\nYes, v4 will ideally have only cosmetic changes apart from the\npart that fixes the traces of the published failed tests:\n\n-- snipsnap --\n15: f678b105f81e ! 15: 7b74987d72a6 tests: include detailed trace logs with --write-junit-xml upon failure\n @@ -32,6 +32,38 @@\n \n Signed-off-by: Johannes Schindelin \n \n + diff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\n + --- a/t/helper/test-path-utils.c\n + +++ b/t/helper/test-path-utils.c\n +@@\n + \t\treturn !!res;\n + \t}\n + \n ++\tif (argc == 4 && !strcmp(argv[1], \"skip-n-bytes\")) {\n ++\t\tint fd = open(argv[2], O_RDONLY), offset = atoi(argv[3]);\n ++\t\tchar buffer[65536];\n ++\n ++\t\tif (fd < 0)\n ++\t\t\tdie_errno(\"could not open '%s'\", argv[2]);\n ++\t\tif (lseek(fd, offset, SEEK_SET) < 0)\n ++\t\t\tdie_errno(\"could not skip %d bytes\", offset);\n ++\t\tfor (;;) {\n ++\t\t\tssize_t count = read(fd, buffer, sizeof(buffer));\n ++\t\t\tif (count < 0)\n ++\t\t\t\tdie_errno(\"could not read '%s'\", argv[2]);\n ++\t\t\tif (!count)\n ++\t\t\t\tbreak;\n ++\t\t\tif (write(1, buffer, count) < 0)\n ++\t\t\t\tdie_errno(\"could not write to stdout\");\n ++\t\t}\n ++\t\tclose(fd);\n ++\t\treturn 0;\n ++\t}\n ++\n + \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n + \t\targv[1] ? argv[1] : \"(there was none)\");\n + \treturn 1;\n +\n diff --git a/t/test-lib.sh b/t/test-lib.sh\n --- a/t/test-lib.sh\n +++ b/t/test-lib.sh\n @@ -42,7 +74,8 @@\n -\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n +\t\t\t\"$(if test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n +\t\t\t then\n -+\t\t\t\tcut -c \"$GIT_TEST_TEE_OFFSET-\" <\"$GIT_TEST_TEE_OUTPUT_FILE\"\n ++\t\t\t\ttest-tool path-utils skip-n-bytes \\\n ++\t\t\t\t\t\"$GIT_TEST_TEE_OUTPUT_FILE\" $GIT_TEST_TEE_OFFSET\n +\t\t\t else\n +\t\t\t\tprintf '%s\\n' \"$@\" | sed 1d\n +\t\t\t fi)\")\"\n @@ -56,17 +89,17 @@\n \tfi\n \ttest_failure=$(($test_failure + 1))\n @@\n - \twrite_junit_xml \"$(printf '%s\\n' \\\n - \t\t\" \" \"$@\" \" \")\"\n - \tjunit_have_testcase=t\n -+\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n + \techo >&3 \"\"\n + \tmaybe_teardown_valgrind\n + \tmaybe_teardown_verbose\n ++\tif test -n \"$GIT_TEST_TEE_OFFSET\"\n +\tthen\n +\t\tGIT_TEST_TEE_OFFSET=$(test-tool path-utils file-size \\\n +\t\t\t\"$GIT_TEST_TEE_OUTPUT_FILE\")\n +\tfi\n }\n \n - test_done () {\n + test_skip () {\n @@\n \t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n \twrite_junit_xml --truncate \"\" \" \"\n @@ -74,7 +107,6 @@\n +\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n +\tthen\n +\t\tGIT_TEST_TEE_OFFSET=0\n -+\t\tGIT_TEST_TEE_ERR_OFFSET=0\n +\tfi\n fi\n \n"},{"id":"367417","messageId":"b086f366606d97097007578ba2ad1523827a4dd7.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 02/21] ci: rename the library of common functions","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:15Z","receivedAt":"2019-01-23T14:40:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe name is hard-coded to reflect that we use Travis CI for continuous\ntesting.\n\nIn the next commits, we will extend this to be able use Azure DevOps,\ntoo.\n\nSo let's adjust the name to make it more generic.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 2 +-\n ci/{lib-travisci.sh => lib.sh} | 0\n ci/print-test-failures.sh | 2 +-\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 2 +-\n 8 files changed, 7 insertions(+), 7 deletions(-)\n rename ci/{lib-travisci.sh => lib.sh} (100%)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 06c3546e1e..fe65144152 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -3,7 +3,7 @@\n # Install dependencies required to build and test Git on Linux and macOS\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\ndiff --git a/ci/lib-travisci.sh b/ci/lib.sh\nsimilarity index 100%\nrename from ci/lib-travisci.sh\nrename to ci/lib.sh\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex d55460a212..7aef39a2fd 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -3,7 +3,7 @@\n # Print output of failing tests\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n # Tracing executed commands would produce too much noise in the loop below.\n set +x\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex cda170d5c2..db342bb6a8 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -3,7 +3,7 @@\n # Build and test Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n ln -s \"$cache_dir/.prove\" t/.prove\n \ndiff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh\nindex 21637903ce..751acfcf8a 100755\n--- a/ci/run-linux32-docker.sh\n+++ b/ci/run-linux32-docker.sh\n@@ -3,7 +3,7 @@\n # Download and run Docker image to build and test 32-bit Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n docker pull daald/ubuntu32:xenial\n \ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\nindex 5688f261d0..dc189c7456 100755\n--- a/ci/run-static-analysis.sh\n+++ b/ci/run-static-analysis.sh\n@@ -3,7 +3,7 @@\n # Perform various static code analysis checks\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n make --jobs=2 coccicheck\n \ndiff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh\nindex d99a180e52..a73a4eca0a 100755\n--- a/ci/run-windows-build.sh\n+++ b/ci/run-windows-build.sh\n@@ -6,7 +6,7 @@\n # supported) and a commit hash.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n test $# -ne 2 && echo \"Unexpected number of parameters\" && exit 1\n test -z \"$GFW_CI_TOKEN\" && echo \"GFW_CI_TOKEN not defined\" && exit\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex a20de9ca12..d3cdbac73f 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -3,7 +3,7 @@\n # Perform sanity checks on documentation and build it.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n gem install asciidoctor\n \n-- \ngitgitgadget\n\n"},{"id":"367418","messageId":"2d0b62f1862d5fce3ac0d7b813399f560600917d.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:16Z","receivedAt":"2019-01-23T14:40:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe upcoming patches will allow building git.git via Azure Pipelines\n(i.e. Azure DevOps' Continuous Integration), where variable names and\nURLs look a bit different than in Travis CI.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 3 ++-\n ci/lib.sh | 41 ++++++++++++++++++++++++++------------\n ci/print-test-failures.sh | 2 +-\n ci/test-documentation.sh | 1 +\n 4 files changed, 32 insertions(+), 15 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex fe65144152..bcdcc71592 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -37,7 +37,8 @@ osx-clang|osx-gcc)\n \tbrew update --quiet\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\n-\tbrew install git-lfs gettext\n+\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n+\tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \tbrew install caskroom/cask/perforce\n \t;;\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex c26bb6a274..4456dbbcb0 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -1,8 +1,26 @@\n # Library of functions shared by all CI scripts\n \n-# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n-# want here. We want the source branch instead.\n-CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+if test true = \"$TRAVIS\"\n+then\n+\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n+\t# what we want here. We want the source branch instead.\n+\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n+\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n+\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n+\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n+\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n+\n+\tcache_dir=\"$HOME/travis-cache\"\n+\n+\turl_for_job_id () {\n+\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+fi\n \n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n@@ -28,7 +46,7 @@ skip_branch_tip_with_tag () {\n # job if we encounter the same tree again and can provide a useful info\n # message.\n save_good_tree () {\n-\techo \"$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID\" >>\"$good_trees_file\"\n+\techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n \t# limit the file size\n \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n \tmv \"$good_trees_file\".tmp \"$good_trees_file\"\n@@ -38,7 +56,7 @@ save_good_tree () {\n # successfully before (e.g. because the branch got rebased, changing only\n # the commit messages).\n skip_good_tree () {\n-\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $TRAVIS_COMMIT^{tree}) \" \"$good_trees_file\")\"\n+\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n \tthen\n \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n \t\t# Continue the build job.\n@@ -48,18 +66,18 @@ skip_good_tree () {\n \techo \"$good_tree_info\" | {\n \t\tread tree prev_good_commit prev_good_job_number prev_good_job_id\n \n-\t\tif test \"$TRAVIS_JOB_ID\" = \"$prev_good_job_id\"\n+\t\tif test \"$CI_JOB_ID\" = \"$prev_good_job_id\"\n \t\tthen\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit has already been built and tested successfully by this build job.\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\telse\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.\n-\t\t\tThe log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id\n+\t\t\tThe log of that build job is available at $(url_for_job_id $prev_good_job_id)\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\tfi\n@@ -84,7 +102,6 @@ check_unignored_build_artifacts ()\n # and installing dependencies.\n set -ex\n \n-cache_dir=\"$HOME/travis-cache\"\n good_trees_file=\"$cache_dir/good-trees\"\n \n mkdir -p \"$cache_dir\"\n@@ -94,13 +111,11 @@ skip_good_tree\n \n if test -z \"$jobname\"\n then\n-\tjobname=\"$TRAVIS_OS_NAME-$CC\"\n+\tjobname=\"$CI_OS_NAME-$CC\"\n fi\n \n export DEVELOPER=1\n export DEFAULT_TEST_TARGET=prove\n-export GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n-export GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n export GIT_TEST_CLONE_2GB=YesPlease\n if [ \"$jobname\" = linux-gcc ]; then\n \texport CC=gcc-8\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 7aef39a2fd..d2045b63a6 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -69,7 +69,7 @@ do\n \tfi\n done\n \n-if [ $combined_trash_size -gt 0 ]\n+if [ -n \"$TRAVIS_JOB_ID\" -a $combined_trash_size -gt 0 ]\n then\n \techo \"------------------------------------------------------------------------\"\n \techo \"Trash directories embedded in this log can be extracted by running:\"\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex d3cdbac73f..7d0beb2832 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -5,6 +5,7 @@\n \n . ${0%/*}/lib.sh\n \n+test -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n gem install asciidoctor\n \n make check-builtins\n-- \ngitgitgadget\n\n"},{"id":"367419","messageId":"6c9eb4e33e1e98210ed5efe7b52fe1a827b2c3ac.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 05/21] ci: use a junction on Windows instead of a symlink","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:17Z","receivedAt":"2019-01-23T14:40:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nSymbolic links are still not quite as easy to use on Windows as on Linux\n(for example, on versions older than Windows 10, only administrators can\ncreate symlinks, and on Windows 10 you still need to be in developer\nmode for regular users to have permission), but NTFS junctions can give\nus a way out.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/run-build-and-tests.sh | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 80d72d120f..74d838ea01 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -5,7 +5,10 @@\n \n . ${0%/*}/lib.sh\n \n-ln -s \"$cache_dir/.prove\" t/.prove\n+case \"$CI_OS_NAME\" in\n+windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n+*) ln -s \"$cache_dir/.prove\" t/.prove;;\n+esac\n \n make\n make --quiet test\n-- \ngitgitgadget\n\n"},{"id":"367420","messageId":"83b92a87e7698cee1e2c44252b934ad436d75a90.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 04/21] ci: inherit --jobs via MAKEFLAGS in run-build-and-tests","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:16Z","receivedAt":"2019-01-23T14:40:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nLet's not decide in the generic ci/ script how many jobs to run in\nparallel; it is easy enough to hand that information down via the\n`MAKEFLAGS`.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/run-build-and-tests.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex db342bb6a8..80d72d120f 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -7,7 +7,7 @@\n \n ln -s \"$cache_dir/.prove\" t/.prove\n \n-make --jobs=2\n+make\n make --quiet test\n if test \"$jobname\" = \"linux-gcc\"\n then\n-- \ngitgitgadget\n\n"},{"id":"367421","messageId":"aa053ed9936ebae0ca5e18d27de96f1d054d7f89.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 06/21] test-date: add a subcommand to measure times in shell scripts","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:18Z","receivedAt":"2019-01-23T14:40:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIn the next commit, we want to teach Git's test suite to optionally\noutput test results in JUnit-style .xml files. These files contain\ninformation about the time spent. So we need a way to measure time.\n\nWhile we could use `date +%s` for that, this will give us only seconds,\ni.e. very coarse-grained timings.\n\nGNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output),\nbut there is no equivalent in BSD `date` (read: on macOS, we would not\nbe able to obtain precise timings).\n\nSo let's introduce `test-tool date getnanos`, with an optional start\ntime, that outputs preciser values.\n\nGranted, it is a bit pointless to try measuring times accurately in\nshell scripts, certainly to nanosecond precision. But it is better than\nsecond-granularity.\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-date.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex a0837371ab..792a805374 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -7,6 +7,7 @@ static const char *usage_msg = \"\\n\"\n \" test-tool date parse [date]...\\n\"\n \" test-tool date approxidate [date]...\\n\"\n \" test-tool date timestamp [date]...\\n\"\n+\" test-tool date getnanos [start-nanos]\\n\"\n \" test-tool date is64bit\\n\"\n \" test-tool date time_t-is64bit\\n\";\n \n@@ -82,6 +83,15 @@ static void parse_approx_timestamp(const char **argv, struct timeval *now)\n \t}\n }\n \n+static void getnanos(const char **argv, struct timeval *now)\n+{\n+\tdouble seconds = getnanotime() / 1.0e9;\n+\n+\tif (*argv)\n+\t\tseconds -= strtod(*argv, NULL);\n+\tprintf(\"%lf\\n\", seconds);\n+}\n+\n int cmd__date(int argc, const char **argv)\n {\n \tstruct timeval now;\n@@ -108,6 +118,8 @@ int cmd__date(int argc, const char **argv)\n \t\tparse_approxidate(argv+1, &now);\n \telse if (!strcmp(*argv, \"timestamp\"))\n \t\tparse_approx_timestamp(argv+1, &now);\n+\telse if (!strcmp(*argv, \"getnanos\"))\n+\t\tgetnanos(argv+1, &now);\n \telse if (!strcmp(*argv, \"is64bit\"))\n \t\treturn sizeof(timestamp_t) == 8 ? 0 : 1;\n \telse if (!strcmp(*argv, \"time_t-is64bit\"))\n-- \ngitgitgadget\n\n"},{"id":"367422","messageId":"eaba471b89976b51ea19fc6772890304135ab539.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 09/21] Add a build definition for Azure DevOps","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:20Z","receivedAt":"2019-01-23T14:40:27Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis commit adds an azure-pipelines.yml file which is Azure DevOps'\nequivalent to Travis CI's .travis.yml.\n\nTo make things a bit easier to understand, we refrain from using the\n`matrix` feature here because (while it is powerful) it can be a bit\nconfusing to users who are not familiar with CI setups. Therefore, we\nuse a separate phase even for similar configurations (such as GCC vs\nClang on Linux, GCC vs Clang on macOS).\n\nAlso, we make use of the shiny new feature we just introduced where the\ntest suite can output JUnit-style .xml files. This information is made\navailable in a nice UI that allows the viewer to filter by phase and/or\ntest number, and to see trends such as: number of (failing) tests, time\nspent running the test suite, etc.\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 306 ++++++++++++++++++++++++++++++++++++++++++\n ci/lib.sh | 2 +-\n ci/mount-fileshare.sh | 25 ++++\n 3 files changed, 332 insertions(+), 1 deletion(-)\n create mode 100644 azure-pipelines.yml\n create mode 100755 ci/mount-fileshare.sh\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nnew file mode 100644\nindex 0000000000..f3cabb0dd0\n--- /dev/null\n+++ b/azure-pipelines.yml\n@@ -0,0 +1,306 @@\n+resources:\n+- repo: self\n+ fetchDepth: 1\n+\n+jobs:\n+- job: linux_clang\n+ displayName: linux-clang\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin &&\n+\n+ export CC=clang || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-clang'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: linux_gcc\n+ displayName: linux-gcc\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo add-apt-repository ppa:ubuntu-toolchain-r/test &&\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2 language-pack-is git-svn gcc-8 || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-gcc'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: osx_clang\n+ displayName: osx-clang\n+ condition: succeeded()\n+ pool: Hosted macOS\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ export CC=clang\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-clang'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: osx_gcc\n+ displayName: osx-gcc\n+ condition: succeeded()\n+ pool: Hosted macOS\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-gcc'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: gettext_poison\n+ displayName: GETTEXT_POISON\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev &&\n+\n+ export jobname=GETTEXT_POISON || exit 1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'gettext-poison'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: windows\n+ displayName: Windows\n+ condition: succeeded()\n+ pool: Hosted\n+ timeoutInMinutes: 240\n+ steps:\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n+ }\n+ displayName: 'Mount test-cache'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - powershell: |\n+ # Helper to check the error level of the latest command (exit with error when appropriate)\n+ function c() { if (!$?) { exit(1) } }\n+\n+ # Add build agent's MinGit to PATH\n+ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n+\n+ # Helper to initialize (or update) a Git worktree\n+ function init ($path, $url, $set_origin) {\n+ if (Test-Path $path) {\n+ cd $path; c\n+ if (Test-Path .git) {\n+ & git init; c\n+ } else {\n+ & git status\n+ }\n+ } else {\n+ & git init $path; c\n+ cd $path; c\n+ }\n+ & git config core.autocrlf false; c\n+ & git config core.untrackedCache true; c\n+ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n+ & git remote add origin $url; c\n+ }\n+ & git fetch --depth=1 $url master; c\n+ & git reset --hard FETCH_HEAD; c\n+ & git clean -df; c\n+ }\n+\n+ # Initialize Git for Windows' SDK\n+ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n+ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+\n+ # Let Git ignore the SDK and the test-cache\n+ \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n+ displayName: 'Initialize the Git for Windows SDK'\n+ - powershell: |\n+ & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ export MAKEFLAGS=-j10\n+ export DEVELOPER=1\n+ export NO_PERL=1\n+ export NO_SVN_TESTS=1\n+ export GIT_TEST_SKIP_REBASE_P=1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+ \"@\n+ if (!$?) { exit(1) }\n+ displayName: 'Build & Test'\n+ env:\n+ HOME: $(Build.SourcesDirectory)\n+ MSYSTEM: MINGW64\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'Unmount test-cache'\n+ condition: true\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'windows'\n+ platform: Windows\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: linux32\n+ displayName: Linux32\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo AGENT_OS=\"$AGENT_OS\" BUILD_BUILDNUMBER=\"$BUILD_BUILDNUMBER\" BUILD_REPOSITORY_URI=\"$BUILD_REPOSITORY_URI\" BUILD_SOURCEBRANCH=\"$BUILD_SOURCEBRANCH\" BUILD_SOURCEVERSION=\"$BUILD_SOURCEVERSION\" SYSTEM_PHASENAME=\"$SYSTEM_PHASENAME\" SYSTEM_TASKDEFINITIONSURI=\"$SYSTEM_TASKDEFINITIONSURI\" SYSTEM_TEAMPROJECT=\"$SYSTEM_TEAMPROJECT\" CC=$CC MAKEFLAGS=-j3 bash -lxc ci/run-linux32-docker.sh || exit 1\n+\n+ sudo chmod a+r t/out/TEST-*.xml\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-linux32-docker.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux32'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n+- job: static_analysis\n+ displayName: StaticAnalysis\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get install -y coccinelle &&\n+\n+ export jobname=StaticAnalysis &&\n+\n+ ci/run-static-analysis.sh || exit 1\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-static-analysis.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+\n+- job: documentation\n+ displayName: Documentation\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get install -y asciidoc xmlto asciidoctor &&\n+\n+ export ALREADY_HAVE_ASCIIDOCTOR=yes. &&\n+ export jobname=Documentation &&\n+\n+ ci/test-documentation.sh || exit 1\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/test-documentation.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex a9eb4f4eae..91cf1402bf 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -42,7 +42,7 @@ then\n \n \tBREW_INSTALL_PACKAGES=\n \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n-\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n fi\n \n skip_branch_tip_with_tag () {\ndiff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\nnew file mode 100755\nindex 0000000000..26b58a8096\n--- /dev/null\n+++ b/ci/mount-fileshare.sh\n@@ -0,0 +1,25 @@\n+#!/bin/sh\n+\n+die () {\n+\techo \"$*\" >&2\n+\texit 1\n+}\n+\n+test $# = 4 ||\n+die \"Usage: $0 \"\n+\n+mkdir -p \"$4\" || die \"Could not create $4\"\n+\n+case \"$(uname -s)\" in\n+Linux)\n+\tsudo mount -t cifs -o vers=3.0,username=\"$2\",password=\"$3\",dir_mode=0777,file_mode=0777,serverino \"$1\" \"$4\"\n+\t;;\n+Darwin)\n+\tpass=\"$(echo \"$3\" | sed -e 's/\\//%2F/g' -e 's/+/%2B/g')\" &&\n+\tmount -t smbfs,soft \"smb://$2:$pass@${1#//}\" \"$4\"\n+\t;;\n+*)\n+\tdie \"No support for $(uname -s)\"\n+\t;;\n+esac ||\n+die \"Could not mount $4\"\n-- \ngitgitgadget\n\n"},{"id":"367423","messageId":"93eebb74ce1e5a2dcacee40c5e1750979bd83b10.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 11/21] ci: use git-sdk-64-minimal build artifact","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:22Z","receivedAt":"2019-01-23T14:40:29Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nInstead of a shallow fetch followed by a sparse checkout, we are\nbetter off by using a separate, dedicated Pipeline that bundles\nthe SDK as a build artifact, and then consuming that build artifact\nhere.\n\nIn fact, since this artifact will be used a lot, we spent substantial\ntime on figuring out a minimal subset of the Git for Windows SDK, just\nenough to build and test Git. The result is a size reduction from around\n1GB (compressed) to around 55MB (compressed). This also comes with the\nchange where we now call `usr\\bin\\bash.exe` directly, as `git-cmd.exe`\nis not included in the minimal SDK.\n\nThat reduces the time to initialize Git for Windows' SDK from anywhere\nbetween 2m30s-7m to a little over 1m.\n\nNote: in theory, we could also use the DownloadBuildArtifacts@0 task\nhere. However, restricted permissions that are in effect when building\nfrom forks would let this fail for PR builds, defeating the whole\npurpose of the Azure Pipelines support for git.git.\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 44 +++++++++-----------------------------------\n 1 file changed, 9 insertions(+), 35 deletions(-)\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex e44d2733a4..3085ff6ace 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -18,44 +18,18 @@ jobs:\n env:\n GITFILESHAREPWD: $(gitfileshare.pwd)\n - powershell: |\n- # Helper to check the error level of the latest command (exit with error when appropriate)\n- function c() { if (!$?) { exit(1) } }\n-\n- # Add build agent's MinGit to PATH\n- $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n-\n- # Helper to initialize (or update) a Git worktree\n- function init ($path, $url, $set_origin) {\n- if (Test-Path $path) {\n- cd $path; c\n- if (Test-Path .git) {\n- & git init; c\n- } else {\n- & git status\n- }\n- } else {\n- & git init $path; c\n- cd $path; c\n- }\n- & git config core.autocrlf false; c\n- & git config core.untrackedCache true; c\n- if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n- & git remote add origin $url; c\n- }\n- & git fetch --depth=1 $url master; c\n- & git reset --hard FETCH_HEAD; c\n- & git clean -df; c\n- }\n-\n- # Initialize Git for Windows' SDK\n- $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n- init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+ $urlbase = \"https://dev.azure.com/git-for-windows/git/_apis/build/builds\"\n+ $id = ((Invoke-WebRequest -UseBasicParsing \"${urlbase}?definitions=22&statusFilter=completed&resultFilter=succeeded&`$top=1\").content | ConvertFrom-JSON).value[0].id\n+ $downloadUrl = ((Invoke-WebRequest -UseBasicParsing \"${urlbase}/$id/artifacts\").content | ConvertFrom-JSON).value[1].resource.downloadUrl\n+ (New-Object Net.WebClient).DownloadFile($downloadUrl,\"git-sdk-64-minimal.zip\")\n+ Expand-Archive git-sdk-64-minimal.zip -DestinationPath . -Force\n+ Remove-Item git-sdk-64-minimal.zip\n \n # Let Git ignore the SDK and the test-cache\n- \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n- displayName: 'Initialize the Git for Windows SDK'\n+ \"/git-sdk-64-minimal/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n+ displayName: 'Download git-sdk-64-minimal'\n - powershell: |\n- & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n export MAKEFLAGS=-j10\n export DEVELOPER=1\n export NO_PERL=1\n-- \ngitgitgadget\n\n"},{"id":"367424","messageId":"3083041e33a3b0c63af621e9b6bc704e288c401f.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 14/21] tests: avoid calling Perl just to determine file sizes","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:25Z","receivedAt":"2019-01-23T14:40:30Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIt is a bit ridiculous to spin up a full-blown Perl instance (especially\non Windows, where that means spinning up a full POSIX emulation layer,\nAKA the MSYS2 runtime) just to tell how large a given file is.\n\nSo let's just use the test-tool to do that job instead.\n\nThis command will also be used over the next commits, to allow for\ncutting out individual test cases' verbose log from the file generated\nvia --verbose-log.\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-path-utils.c | 12 ++++++++++++\n t/t0021-conversion.sh | 2 +-\n t/t1050-large.sh | 2 +-\n t/t5315-pack-objects-compression.sh | 2 +-\n t/t9303-fast-import-compression.sh | 2 +-\n 5 files changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex ae091d9b3e..30211d6d64 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -291,6 +291,18 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn !!res;\n \t}\n \n+\tif (argc > 2 && !strcmp(argv[1], \"file-size\")) {\n+\t\tint res = 0, i;\n+\t\tstruct stat st;\n+\n+\t\tfor (i = 2; i < argc; i++)\n+\t\t\tif (stat(argv[i], &st))\n+\t\t\t\tres = error_errno(\"Cannot stat '%s'\", argv[i]);\n+\t\t\telse\n+\t\t\t\tprintf(\"%\"PRIuMAX\"\\n\", (uintmax_t)st.st_size);\n+\t\treturn !!res;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex fd5f1ac649..e10f5f787f 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -24,7 +24,7 @@ generate_random_characters () {\n }\n \n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n filter_git () {\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 1a9b21b293..dcb4dbba67 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -8,7 +8,7 @@ test_description='adding and checking out large blobs'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n test_expect_success setup '\ndiff --git a/t/t5315-pack-objects-compression.sh b/t/t5315-pack-objects-compression.sh\nindex 34c47dae09..df970d7584 100755\n--- a/t/t5315-pack-objects-compression.sh\n+++ b/t/t5315-pack-objects-compression.sh\n@@ -7,7 +7,7 @@ test_description='pack-object compression configuration'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n test_expect_success setup '\ndiff --git a/t/t9303-fast-import-compression.sh b/t/t9303-fast-import-compression.sh\nindex 856219f46a..5045f02a53 100755\n--- a/t/t9303-fast-import-compression.sh\n+++ b/t/t9303-fast-import-compression.sh\n@@ -6,7 +6,7 @@ test_description='compression setting of fast-import utility'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n import_large () {\n-- \ngitgitgadget\n\n"},{"id":"367425","messageId":"1c44d3b8b81946c66839b7e750da2336a7ec4ad3.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 15/21] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:26Z","receivedAt":"2019-01-23T14:40:32Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe JUnit XML format lends itself to be presented in a powerful UI,\nwhere you can drill down to the information you are interested in very\nquickly.\n\nFor test failures, this usually means that you want to see the detailed\ntrace of the failing tests.\n\nWith Travis CI, we passed the `--verbose-log` option to get those\ntraces. However, that seems excessive, as we do not need/use the logs in\nalmost all of those cases: only when a test fails do we have a way to\ninclude the trace.\n\nSo let's do something different when using Azure DevOps: let's run all\nthe tests with `--quiet` first, and only if a failure is encountered,\ntry to trace the commands as they are executed.\n\nOf course, we cannot turn on `--verbose-log` after the fact. So let's\njust re-run the test with all the same options, adding `--verbose-log`.\nAnd then munging the output file into the JUnit XML on the fly.\n\nNote: there is an off chance that re-running the test in verbose mode\n\"fixes\" the failures (and this does happen from time to time!). That is\na possibility we should be able to live with. Ideally, we would label\nthis as \"Passed upon rerun\", and Azure Pipelines even know about that\noutcome, but it is not available when using the JUnit XML format for\nnow:\nhttps://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-path-utils.c | 21 +++++++++++++++++++++\n t/test-lib.sh | 22 +++++++++++++++++++++-\n 2 files changed, 42 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex 30211d6d64..6efde6f5ba 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -303,6 +303,27 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn !!res;\n \t}\n \n+\tif (argc == 4 && !strcmp(argv[1], \"skip-n-bytes\")) {\n+\t\tint fd = open(argv[2], O_RDONLY), offset = atoi(argv[3]);\n+\t\tchar buffer[65536];\n+\n+\t\tif (fd < 0)\n+\t\t\tdie_errno(\"could not open '%s'\", argv[2]);\n+\t\tif (lseek(fd, offset, SEEK_SET) < 0)\n+\t\t\tdie_errno(\"could not skip %d bytes\", offset);\n+\t\tfor (;;) {\n+\t\t\tssize_t count = read(fd, buffer, sizeof(buffer));\n+\t\t\tif (count < 0)\n+\t\t\t\tdie_errno(\"could not read '%s'\", argv[2]);\n+\t\t\tif (!count)\n+\t\t\t\tbreak;\n+\t\t\tif (write(1, buffer, count) < 0)\n+\t\t\t\tdie_errno(\"could not write to stdout\");\n+\t\t}\n+\t\tclose(fd);\n+\t\treturn 0;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a3b2166cb5..f31a1c8f79 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -639,8 +639,19 @@ test_failure_ () {\n \t\tjunit_insert=\"\"\n \t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n-\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\t\t\"$(if test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\t\t then\n+\t\t\t\ttest-tool path-utils skip-n-bytes \\\n+\t\t\t\t\t\"$GIT_TEST_TEE_OUTPUT_FILE\" $GIT_TEST_TEE_OFFSET\n+\t\t\t else\n+\t\t\t\tprintf '%s\\n' \"$@\" | sed 1d\n+\t\t\t fi)\")\"\n \t\tjunit_insert=\"$junit_insert\"\n+\t\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\tthen\n+\t\t\tjunit_insert=\"$junit_insert$(xml_attr_encode \\\n+\t\t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n+\t\tfi\n \t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n \tfi\n \ttest_failure=$(($test_failure + 1))\n@@ -931,6 +942,11 @@ test_finish_ () {\n \techo >&3 \"\"\n \tmaybe_teardown_valgrind\n \tmaybe_teardown_verbose\n+\tif test -n \"$GIT_TEST_TEE_OFFSET\"\n+\tthen\n+\t\tGIT_TEST_TEE_OFFSET=$(test-tool path-utils file-size \\\n+\t\t\t\"$GIT_TEST_TEE_OUTPUT_FILE\")\n+\tfi\n }\n \n test_skip () {\n@@ -1280,6 +1296,10 @@ then\n \t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n \twrite_junit_xml --truncate \"\" \" \"\n \tjunit_suite_start=$(test-tool date getnanos)\n+\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\tthen\n+\t\tGIT_TEST_TEE_OFFSET=0\n+\tfi\n fi\n \n # Provide an implementation of the 'yes' utility\n-- \ngitgitgadget\n\n"},{"id":"367426","messageId":"a198885b76cc34d636ff2d746123c9cc9ae6ab27.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 19/21] tests: optionally skip bin-wrappers/","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:29Z","receivedAt":"2019-01-23T14:40:34Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis speeds up the tests by a bit on Windows, where running Unix shell\nscripts (and spawning processes) is not exactly a cheap operation.\n\nSigned-off-by: Johannes Schindelin \n---\n t/README | 9 +++++++++\n t/test-lib.sh | 19 +++++++++++++------\n 2 files changed, 22 insertions(+), 6 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 11ce7675e3..063530234f 100644\n--- a/t/README\n+++ b/t/README\n@@ -170,6 +170,15 @@ appropriately before running \"make\".\n \timplied by other options like --valgrind and\n \tGIT_TEST_INSTALLED.\n \n+--no-bin-wrappers::\n+\tBy default, the test suite uses the wrappers in\n+\t`../bin-wrappers/` to execute `git` and friends. With this option,\n+\t`../git` and friends are run directly. This is not recommended\n+\tin general, as the wrappers contain safeguards to ensure that no\n+\tfiles from an installed Git are used, but can speed up test runs\n+\tespecially on platforms where running shell scripts is expensive\n+\t(most notably, Windows).\n+\n --root=::\n \tCreate \"trash\" directories used to store all temporary data during\n \ttesting under , instead of the t/ directory.\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex c790e98fd2..25e649c997 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -111,6 +111,8 @@ do\n \t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n \t--with-dashes)\n \t\twith_dashes=t ;;\n+\t--no-bin-wrappers)\n+\t\tno_bin_wrappers=t ;;\n \t--no-color)\n \t\tcolor= ;;\n \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n@@ -1214,16 +1216,21 @@ then\n \tPATH=$GIT_TEST_INSTALLED:$GIT_BUILD_DIR/t/helper:$PATH\n \tGIT_EXEC_PATH=${GIT_TEST_EXEC_PATH:-$GIT_EXEC_PATH}\n else # normal case, use ../bin-wrappers only unless $with_dashes:\n-\tgit_bin_dir=\"$GIT_BUILD_DIR/bin-wrappers\"\n-\tif ! test -x \"$git_bin_dir/git\"\n+\tif test -n \"$no_bin_wrappers\"\n \tthen\n-\t\tif test -z \"$with_dashes\"\n+\t\twith_dashes=t\n+\telse\n+\t\tgit_bin_dir=\"$GIT_BUILD_DIR/bin-wrappers\"\n+\t\tif ! test -x \"$git_bin_dir/git\"\n \t\tthen\n-\t\t\tsay \"$git_bin_dir/git is not executable; using GIT_EXEC_PATH\"\n+\t\t\tif test -z \"$with_dashes\"\n+\t\t\tthen\n+\t\t\t\tsay \"$git_bin_dir/git is not executable; using GIT_EXEC_PATH\"\n+\t\t\tfi\n+\t\t\twith_dashes=t\n \t\tfi\n-\t\twith_dashes=t\n+\t\tPATH=\"$git_bin_dir:$PATH\"\n \tfi\n-\tPATH=\"$git_bin_dir:$PATH\"\n \tGIT_EXEC_PATH=$GIT_BUILD_DIR\n \tif test -n \"$with_dashes\"\n \tthen\n-- \ngitgitgadget\n\n"},{"id":"367428","messageId":"d613c79aff9c65ab9c3723a876c99e6f35a2f133.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 18/21] t0061: fix with --with-dashes and RUNTIME_PREFIX","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:29Z","receivedAt":"2019-01-23T14:40:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWhen building Git with RUNTIME_PREFIX and starting a test helper from\nt/helper/, it fails to detect the system prefix correctly.\n\nThis is the reason that the warning\n\n\tRUNTIME_PREFIX requested, but prefix computation failed. [...]\n\nto be printed.\n\nIn t0061, we did not expect that to happen, and it actually did not\nhappen in the normal case, because bin-wrappers/test-tool specifically\nsets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool wants\nto know about the runtime prefix).\n\nHowever, with --with-dashes, bin-wrappers/test-tool is no longer called,\nbut t/helper/test-tool is called directly.\n\nSo let's just ignore the RUNTIME_PREFIX warning.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t0061-run-command.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 99a614bc7c..5a2d087bf0 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -166,7 +166,8 @@ test_trace () {\n \texpect=\"$1\"\n \tshift\n \tGIT_TRACE=1 test-tool run-command \"$@\" run-command true 2>&1 >/dev/null | \\\n-\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' >actual &&\n+\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' \\\n+\t\t\t-e '/RUNTIME_PREFIX requested/d' >actual &&\n \techo \"$expect true\" >expect &&\n \ttest_cmp expect actual\n }\n-- \ngitgitgadget\n\n"},{"id":"367429","messageId":"59c1194ae2da0c4986f595ab9600c9dc518ef1b1.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 16/21] mingw: try to work around issues with the test cleanup","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:27Z","receivedAt":"2019-01-23T14:40:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIt seems that every once in a while in the Git for Windows SDK, there\nare some transient file locking issues preventing the test clean up to\ndelete the trash directory. Let's be gentle and try again five seconds\nlater, and only error out if it still fails the second time.\n\nThis change helps Windows, and does not hurt any other platform\n(normally, it is highly unlikely that said deletion fails, and if it\ndoes, normally it will fail again even 5 seconds later).\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex f31a1c8f79..9c0ca5effb 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1104,7 +1104,11 @@ test_done () {\n \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n \n \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n-\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n+\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n+\t\t\t\t# try again in a bit\n+\t\t\t\tsleep 5;\n+\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n+\t\t\t} ||\n \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n \t\tfi\n \t\ttest_at_end_hook_\n-- \ngitgitgadget\n\n"},{"id":"367430","messageId":"966c412f03013902826ffd188ee9af973be60833.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 17/21] tests: add t/helper/ to the PATH with --with-dashes","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:28Z","receivedAt":"2019-01-23T14:40:39Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWe really need to be able to find the test helpers... Really. This\nchange was forgotten when we moved the test helpers into t/helper/\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9c0ca5effb..c790e98fd2 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1227,7 +1227,7 @@ else # normal case, use ../bin-wrappers only unless $with_dashes:\n \tGIT_EXEC_PATH=$GIT_BUILD_DIR\n \tif test -n \"$with_dashes\"\n \tthen\n-\t\tPATH=\"$GIT_BUILD_DIR:$PATH\"\n+\t\tPATH=\"$GIT_BUILD_DIR:$GIT_BUILD_DIR/t/helper:$PATH\"\n \tfi\n fi\n GIT_TEMPLATE_DIR=\"$GIT_BUILD_DIR\"/templates/blt\n-- \ngitgitgadget\n\n"},{"id":"367431","messageId":"6520f4603c78d38f5168164218dc108833a5583d.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 20/21] ci: speed up Windows phase","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:30Z","receivedAt":"2019-01-23T14:40:40Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nAs Unix shell scripting comes at a hefty price on Windows, we have to\nsee where we can save some time to run the test suite.\n\nLet's skip the chain linting and the bin-wrappers/ redirection on\nWindows; this seems to shave of anywhere between 10-30% from the overall\nruntime.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 91cf1402bf..e1d5222ceb 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -43,6 +43,8 @@ then\n \tBREW_INSTALL_PACKAGES=\n \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n+\ttest windows_nt != \"$CI_OS_NAME\" ||\n+\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n fi\n \n skip_branch_tip_with_tag () {\n-- \ngitgitgadget\n\n"},{"id":"367427","messageId":"8bdd9804a1a7751c9646c585022ab4d043eb2168.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 21/21] ci: parallelize testing on Windows","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:31Z","receivedAt":"2019-01-23T14:40:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe fact that Git's test suite is implemented in Unix shell script that\nis as portable as we can muster, combined with the fact that Unix shell\nscripting is foreign to Windows (and therefore has to be emulated),\nresults in pretty abysmal speed of the test suite on that platform, for\npretty much no other reason than that language choice.\n\nFor comparison: while the Linux build & test is typically done within\nabout 8 minutes, the Windows build & test typically lasts about 80\nminutes in Azure Pipelines.\n\nTo help with that, let's use the Azure Pipeline feature where you can\nparallelize jobs, make jobs depend on each other, and pass artifacts\nbetween them.\n\nThe tests are distributed using the following heuristic: listing all\ntest scripts ordered by size in descending order (as a cheap way to\nestimate the overall run time), every Nth script is run (where N is the\ntotal number of parallel jobs), starting at the index corresponding to\nthe parallel job. This slicing is performed by a new function that is\nadded to the `test-tool`.\n\nTo optimize the overall runtime of the entire Pipeline, we need to move\nthe Windows jobs to the beginning (otherwise there would be a very\ndecent chance for the Pipeline to be run only the Windows build, while\nall the parallel Windows test jobs wait for this single one).\n\nWe use Azure Pipelines Artifacts for both the minimal Git for Windows\nSDK as well as the built executables, as deduplication and caching close\nto the agents makes that really fast. For comparison: while downloading\nand unpacking the minimal Git for Windows SDK via PowerShell takes only\none minute (down from anywhere between 2.5 to 7 when using a shallow\nclone), uploading it as Pipeline Artifact takes less than 30s and\ndownloading and unpacking less than 20s (sometimes even as little as\nonly twelve seconds).\n\nSigned-off-by: Johannes Schindelin \n---\n Makefile | 10 +++++\n azure-pipelines.yml | 82 +++++++++++++++++++++++++++++++++-----\n ci/make-test-artifacts.sh | 12 ++++++\n ci/run-test-slice.sh | 17 ++++++++\n t/helper/test-path-utils.c | 31 ++++++++++++++\n 5 files changed, 143 insertions(+), 9 deletions(-)\n create mode 100755 ci/make-test-artifacts.sh\n create mode 100755 ci/run-test-slice.sh\n\ndiff --git a/Makefile b/Makefile\nindex 044b4f77bd..daa318fe17 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2927,6 +2927,16 @@ rpm::\n \t@false\n .PHONY: rpm\n \n+artifacts-tar:: $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) \\\n+\t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n+\t\t$(NO_INSTALL) $(MOFILES)\n+\t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n+\t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n+\ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n+\tmkdir -p \"$(ARTIFACTS_DIRECTORY)\"\n+\t$(TAR) czf \"$(ARTIFACTS_DIRECTORY)/artifacts.tar.gz\" $^ templates/blt/\n+.PHONY: artifacts-tar\n+\n htmldocs = git-htmldocs-$(GIT_VERSION)\n manpages = git-manpages-$(GIT_VERSION)\n .PHONY: dist-doc distclean\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex 3085ff6ace..6cd27b3483 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -3,8 +3,8 @@ resources:\n fetchDepth: 1\n \n jobs:\n-- job: windows\n- displayName: Windows\n+- job: windows_build\n+ displayName: Windows Build\n condition: succeeded()\n pool: Hosted\n timeoutInMinutes: 240\n@@ -30,22 +30,86 @@ jobs:\n displayName: 'Download git-sdk-64-minimal'\n - powershell: |\n & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n- export MAKEFLAGS=-j10\n- export DEVELOPER=1\n- export NO_PERL=1\n- export NO_SVN_TESTS=1\n- export GIT_TEST_SKIP_REBASE_P=1\n+ ci/make-test-artifacts.sh artifacts\n+ \"@\n+ if (!$?) { exit(1) }\n+ displayName: Build\n+ env:\n+ HOME: $(Build.SourcesDirectory)\n+ MSYSTEM: MINGW64\n+ MAKEFLAGS: -j10\n+ DEVELOPER: 1\n+ NO_PERL: 1\n+ - task: PublishPipelineArtifact@0\n+ displayName: 'Publish Pipeline Artifact: test artifacts'\n+ inputs:\n+ artifactName: 'windows-artifacts'\n+ targetPath: '$(Build.SourcesDirectory)\\artifacts'\n+ - task: PublishPipelineArtifact@0\n+ displayName: 'Publish Pipeline Artifact: git-sdk-64-minimal'\n+ inputs:\n+ artifactName: 'git-sdk-64-minimal'\n+ targetPath: '$(Build.SourcesDirectory)\\git-sdk-64-minimal'\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'Unmount test-cache'\n+ condition: true\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+\n+- job: windows_test\n+ displayName: Windows Test\n+ dependsOn: windows_build\n+ condition: succeeded()\n+ pool: Hosted\n+ timeoutInMinutes: 240\n+ strategy:\n+ parallel: 10\n+ steps:\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n+ }\n+ displayName: 'Mount test-cache'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: DownloadPipelineArtifact@0\n+ displayName: 'Download Pipeline Artifact: test artifacts'\n+ inputs:\n+ artifactName: 'windows-artifacts'\n+ targetPath: '$(Build.SourcesDirectory)'\n+ - task: DownloadPipelineArtifact@0\n+ displayName: 'Download Pipeline Artifact: git-sdk-64-minimal'\n+ inputs:\n+ artifactName: 'git-sdk-64-minimal'\n+ targetPath: '$(Build.SourcesDirectory)\\git-sdk-64-minimal'\n+ - powershell: |\n+ & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n+ test -f artifacts.tar.gz || {\n+ echo No test artifacts found\\; skipping >&2\n+ exit 0\n+ }\n+ tar xf artifacts.tar.gz || exit 1\n+\n+ # Let Git ignore the SDK and the test-cache\n+ printf '%s\\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude\n \n- ci/run-build-and-tests.sh || {\n+ ci/run-test-slice.sh `$SYSTEM_JOBPOSITIONINPHASE `$SYSTEM_TOTALJOBSINPHASE || {\n ci/print-test-failures.sh\n exit 1\n }\n \"@\n if (!$?) { exit(1) }\n- displayName: 'Build & Test'\n+ displayName: 'Test (parallel)'\n env:\n HOME: $(Build.SourcesDirectory)\n MSYSTEM: MINGW64\n+ MAKEFLAGS: -j10\n+ NO_SVN_TESTS: 1\n+ GIT_TEST_SKIP_REBASE_P: 1\n - powershell: |\n if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\ndiff --git a/ci/make-test-artifacts.sh b/ci/make-test-artifacts.sh\nnew file mode 100755\nindex 0000000000..646967481f\n--- /dev/null\n+++ b/ci/make-test-artifacts.sh\n@@ -0,0 +1,12 @@\n+#!/bin/sh\n+#\n+# Build Git and store artifacts for testing\n+#\n+\n+mkdir -p \"$1\" # in case ci/lib.sh decides to quit early\n+\n+. ${0%/*}/lib.sh\n+\n+make artifacts-tar ARTIFACTS_DIRECTORY=\"$1\"\n+\n+check_unignored_build_artifacts\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nnew file mode 100755\nindex 0000000000..f8c2c3106a\n--- /dev/null\n+++ b/ci/run-test-slice.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+#\n+# Test Git in parallel\n+#\n+\n+. ${0%/*}/lib.sh\n+\n+case \"$CI_OS_NAME\" in\n+windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n+*) ln -s \"$cache_dir/.prove\" t/.prove;;\n+esac\n+\n+make --quiet -C t T=\"$(cd t &&\n+\t./helper/test-tool path-utils slice-tests \"$1\" \"$2\" t[0-9]*.sh |\n+\ttr '\\n' ' ')\"\n+\n+check_unignored_build_artifacts\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex 6efde6f5ba..5d543ad21f 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -177,6 +177,14 @@ static int is_dotgitmodules(const char *path)\n \treturn is_hfs_dotgitmodules(path) || is_ntfs_dotgitmodules(path);\n }\n \n+static int cmp_by_st_size(const void *a, const void *b)\n+{\n+\tintptr_t x = (intptr_t)((struct string_list_item *)a)->util;\n+\tintptr_t y = (intptr_t)((struct string_list_item *)b)->util;\n+\n+\treturn x > y ? -1 : (x < y ? +1 : 0);\n+}\n+\n int cmd__path_utils(int argc, const char **argv)\n {\n \tif (argc == 3 && !strcmp(argv[1], \"normalize_path_copy\")) {\n@@ -324,6 +332,29 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn 0;\n \t}\n \n+\tif (argc > 5 && !strcmp(argv[1], \"slice-tests\")) {\n+\t\tint res = 0;\n+\t\tlong offset, stride, i;\n+\t\tstruct string_list list = STRING_LIST_INIT_NODUP;\n+\t\tstruct stat st;\n+\n+\t\toffset = strtol(argv[2], NULL, 10);\n+\t\tstride = strtol(argv[3], NULL, 10);\n+\t\tif (stride < 1)\n+\t\t\tstride = 1;\n+\t\tfor (i = 4; i < argc; i++)\n+\t\t\tif (stat(argv[i], &st))\n+\t\t\t\tres = error_errno(\"Cannot stat '%s'\", argv[i]);\n+\t\t\telse\n+\t\t\t\tstring_list_append(&list, argv[i])->util =\n+\t\t\t\t\t(void *)(intptr_t)st.st_size;\n+\t\tQSORT(list.items, list.nr, cmp_by_st_size);\n+\t\tfor (i = offset; i < list.nr; i+= stride)\n+\t\t\tprintf(\"%s\\n\", list.items[i].string);\n+\n+\t\treturn !!res;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\n-- \ngitgitgadget\n"},{"id":"367432","messageId":"18d76823e56547e47be39ad657a1795603659593.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 12/21] mingw: be more generous when wrapping up the setitimer() emulation","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:23Z","receivedAt":"2019-01-23T14:40:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nEvery once in a while, the Azure Pipeline fails with some semi-random\n\n\terror: timer thread did not terminate timely\n\nThis error message means that the thread that is used to emulate the\nsetitimer() function did not terminate within 1,000 milliseconds.\n\nThe most likely explanation (and therefore the one we should assume to\nbe true, according to Occam's Razor) is that the timeout of one second\nis simply not enough because we try to run so many tasks in parallel.\n\nSo let's give it ten seconds instead of only one. That should be enough.\n\nSigned-off-by: Johannes Schindelin \n---\n compat/mingw.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex b459e1a291..e0dfe8844d 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -2100,7 +2100,7 @@ static void stop_timer_thread(void)\n \tif (timer_event)\n \t\tSetEvent(timer_event);\t/* tell thread to terminate */\n \tif (timer_thread) {\n-\t\tint rc = WaitForSingleObject(timer_thread, 1000);\n+\t\tint rc = WaitForSingleObject(timer_thread, 10000);\n \t\tif (rc == WAIT_TIMEOUT)\n \t\t\terror(\"timer thread did not terminate timely\");\n \t\telse if (rc != WAIT_OBJECT_0)\n-- \ngitgitgadget\n\n"},{"id":"367433","messageId":"07fb37e0a6b7382b990739c491a6499e95b25b06.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 07/21] tests: optionally write results as JUnit-style .xml","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:19Z","receivedAt":"2019-01-23T14:40:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis will come in handy when publishing the results of Git's test suite\nduring an automated Azure DevOps run.\n\nNote: we need to make extra sure that invalid UTF-8 encoding is turned\ninto valid UTF-8 (using the Replacement Character, \\uFFFD) because\nt9902's trace contains such invalid byte sequences, and the task in the\nAzure Pipeline that uploads the test results would refuse to do anything\nif it was asked to parse an .xml file with invalid UTF-8 in it.\n\nSigned-off-by: Johannes Schindelin \n---\n Makefile | 1 +\n t/.gitignore | 1 +\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/helper/test-xml-encode.c | 80 +++++++++++++++++++++++++++++++++\n t/test-lib.sh | 91 ++++++++++++++++++++++++++++++++++++++\n 6 files changed, 175 insertions(+)\n create mode 100644 t/helper/test-xml-encode.c\n\ndiff --git a/Makefile b/Makefile\nindex 1a44c811aa..044b4f77bd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -754,6 +754,7 @@ TEST_BUILTINS_OBJS += test-submodule-config.o\n TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o\n TEST_BUILTINS_OBJS += test-subprocess.o\n TEST_BUILTINS_OBJS += test-urlmatch-normalization.o\n+TEST_BUILTINS_OBJS += test-xml-encode.o\n TEST_BUILTINS_OBJS += test-wildmatch.o\n TEST_BUILTINS_OBJS += test-windows-named-pipe.o\n TEST_BUILTINS_OBJS += test-write-cache.o\ndiff --git a/t/.gitignore b/t/.gitignore\nindex 348715f0e4..91cf5772fe 100644\n--- a/t/.gitignore\n+++ b/t/.gitignore\n@@ -2,3 +2,4 @@\n /test-results\n /.prove\n /chainlinttmp\n+/out/\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex bfb195b1a8..4b4b397d93 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -49,6 +49,7 @@ static struct test_cmd cmds[] = {\n \t{ \"submodule-nested-repo-config\", cmd__submodule_nested_repo_config },\n \t{ \"subprocess\", cmd__subprocess },\n \t{ \"urlmatch-normalization\", cmd__urlmatch_normalization },\n+\t{ \"xml-encode\", cmd__xml_encode },\n \t{ \"wildmatch\", cmd__wildmatch },\n #ifdef GIT_WINDOWS_NATIVE\n \t{ \"windows-named-pipe\", cmd__windows_named_pipe },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 042f12464b..c0ab65e370 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -45,6 +45,7 @@ int cmd__submodule_config(int argc, const char **argv);\n int cmd__submodule_nested_repo_config(int argc, const char **argv);\n int cmd__subprocess(int argc, const char **argv);\n int cmd__urlmatch_normalization(int argc, const char **argv);\n+int cmd__xml_encode(int argc, const char **argv);\n int cmd__wildmatch(int argc, const char **argv);\n #ifdef GIT_WINDOWS_NATIVE\n int cmd__windows_named_pipe(int argc, const char **argv);\ndiff --git a/t/helper/test-xml-encode.c b/t/helper/test-xml-encode.c\nnew file mode 100644\nindex 0000000000..367c4875e6\n--- /dev/null\n+++ b/t/helper/test-xml-encode.c\n@@ -0,0 +1,80 @@\n+#include \"test-tool.h\"\n+\n+static const char *utf8_replace_character = \"�\";\n+\n+/*\n+ * Encodes (possibly incorrect) UTF-8 on to , to be embedded\n+ * in an XML file.\n+ */\n+int cmd__xml_encode(int argc, const char **argv)\n+{\n+\tunsigned char buf[1024], tmp[4], *tmp2 = NULL;\n+\tssize_t cur = 0, len = 1, remaining = 0;\n+\tunsigned char ch;\n+\n+\tfor (;;) {\n+\t\tif (++cur == len) {\n+\t\t\tlen = xread(0, buf, sizeof(buf));\n+\t\t\tif (!len)\n+\t\t\t\treturn 0;\n+\t\t\tif (len < 0)\n+\t\t\t\tdie_errno(\"Could not read \");\n+\t\t\tcur = 0;\n+\t\t}\n+\t\tch = buf[cur];\n+\n+\t\tif (tmp2) {\n+\t\t\tif ((ch & 0xc0) != 0x80) {\n+\t\t\t\tfputs(utf8_replace_character, stdout);\n+\t\t\t\ttmp2 = 0;\n+\t\t\t\tcur--;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\t*tmp2 = ch;\n+\t\t\ttmp2++;\n+\t\t\tif (--remaining == 0) {\n+\t\t\t\tfwrite(tmp, tmp2 - tmp, 1, stdout);\n+\t\t\t\ttmp2 = 0;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!(ch & 0x80)) {\n+\t\t\t/* 0xxxxxxx */\n+\t\t\tif (ch == '&')\n+\t\t\t\tfputs(\"&\", stdout);\n+\t\t\telse if (ch == '\\'')\n+\t\t\t\tfputs(\"'\", stdout);\n+\t\t\telse if (ch == '\"')\n+\t\t\t\tfputs(\""\", stdout);\n+\t\t\telse if (ch == '<')\n+\t\t\t\tfputs(\"<\", stdout);\n+\t\t\telse if (ch == '>')\n+\t\t\t\tfputs(\">\", stdout);\n+\t\t\telse if (ch >= 0x20)\n+\t\t\t\tfputc(ch, stdout);\n+\t\t\telse if (ch == 0x09 || ch == 0x0a || ch == 0x0d)\n+\t\t\t\tfprintf(stdout, \"&#x%02x;\", ch);\n+\t\t\telse\n+\t\t\t\tfputs(utf8_replace_character, stdout);\n+\t\t} else if ((ch & 0xe0) == 0xc0) {\n+\t\t\t/* 110XXXXx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 1;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else if ((ch & 0xf0) == 0xe0) {\n+\t\t\t/* 1110XXXX 10Xxxxxx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 2;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else if ((ch & 0xf8) == 0xf0) {\n+\t\t\t/* 11110XXX 10XXxxxx 10xxxxxx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 3;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else\n+\t\t\tfputs(utf8_replace_character, stdout);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a1abb1177a..a3b2166cb5 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -139,6 +139,9 @@ do\n \t\tverbose_log=t\n \t\ttee=t\n \t\t;;\n+\t--write-junit-xml)\n+\t\twrite_junit_xml=t\n+\t\t;;\n \t--stress)\n \t\tstress=t ;;\n \t--stress=*)\n@@ -622,11 +625,24 @@ trap 'exit $?' INT TERM HUP\n # the test_expect_* functions instead.\n \n test_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$*\"\n+\tfi\n \ttest_success=$(($test_success + 1))\n \tsay_color \"\" \"ok $test_count - $@\"\n }\n \n test_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_insert=\"\"\n+\t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n+\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\tjunit_insert=\"$junit_insert\"\n+\t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n+\tfi\n \ttest_failure=$(($test_failure + 1))\n \tsay_color error \"not ok $test_count - $1\"\n \tshift\n@@ -635,11 +651,19 @@ test_failure_ () {\n }\n \n test_known_broken_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (breakage fixed)\"\n+\tfi\n \ttest_fixed=$(($test_fixed+1))\n \tsay_color error \"ok $test_count - $@ # TODO known breakage vanished\"\n }\n \n test_known_broken_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (known breakage)\"\n+\tfi\n \ttest_broken=$(($test_broken+1))\n \tsay_color warn \"not ok $test_count - $@ # TODO known breakage\"\n }\n@@ -897,6 +921,10 @@ test_start_ () {\n \ttest_count=$(($test_count+1))\n \tmaybe_setup_verbose\n \tmaybe_setup_valgrind\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_start=$(test-tool date getnanos)\n+\tfi\n }\n \n test_finish_ () {\n@@ -934,6 +962,13 @@ test_skip () {\n \n \tcase \"$to_skip\" in\n \tt)\n+\t\tif test -n \"$write_junit_xml\"\n+\t\tthen\n+\t\t\tmessage=\"$(xml_attr_encode \"$skipped_reason\")\"\n+\t\t\twrite_junit_xml_testcase \"$1\" \\\n+\t\t\t\t\" \"\n+\t\tfi\n+\n \t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n@@ -949,9 +984,51 @@ test_at_end_hook_ () {\n \t:\n }\n \n+write_junit_xml () {\n+\tcase \"$1\" in\n+\t--truncate)\n+\t\t>\"$junit_xml_path\"\n+\t\tjunit_have_testcase=\n+\t\tshift\n+\t\t;;\n+\tesac\n+\tprintf '%s\\n' \"$@\" >>\"$junit_xml_path\"\n+}\n+\n+xml_attr_encode () {\n+\tprintf '%s\\n' \"$@\" | test-tool xml-encode\n+}\n+\n+write_junit_xml_testcase () {\n+\tjunit_attrs=\"name=\\\"$(xml_attr_encode \"$this_test.$test_count $1\")\\\"\"\n+\tshift\n+\tjunit_attrs=\"$junit_attrs classname=\\\"$this_test\\\"\"\n+\tjunit_attrs=\"$junit_attrs time=\\\"$(test-tool \\\n+\t\tdate getnanos $junit_start)\\\"\"\n+\twrite_junit_xml \"$(printf '%s\\n' \\\n+\t\t\" \" \"$@\" \" \")\"\n+\tjunit_have_testcase=t\n+}\n+\n test_done () {\n \tGIT_EXIT_OK=t\n \n+\tif test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n+\tthen\n+\t\ttest -n \"$junit_have_testcase\" || {\n+\t\t\tjunit_start=$(test-tool date getnanos)\n+\t\t\twrite_junit_xml_testcase \"all tests skipped\"\n+\t\t}\n+\n+\t\t# adjust the overall time\n+\t\tjunit_time=$(test-tool date getnanos $junit_suite_start)\n+\t\tsed \"s/]*/& time=\\\"$junit_time\\\"/\" \\\n+\t\t\t<\"$junit_xml_path\" >\"$junit_xml_path.new\"\n+\t\tmv \"$junit_xml_path.new\" \"$junit_xml_path\"\n+\n+\t\twrite_junit_xml \" \" \"\"\n+\tfi\n+\n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n \t\tmkdir -p \"$TEST_RESULTS_DIR\"\n@@ -1178,6 +1255,7 @@ then\n else\n \tmkdir -p \"$TRASH_DIRECTORY\"\n fi\n+\n # Use -P to resolve symlinks in our working directory so that the cwd\n # in subprocesses like git equals our $PWD (for pathname comparisons).\n cd -P \"$TRASH_DIRECTORY\" || exit 1\n@@ -1191,6 +1269,19 @@ then\n \ttest_done\n fi\n \n+if test -n \"$write_junit_xml\"\n+then\n+\tjunit_xml_dir=\"$TEST_OUTPUT_DIRECTORY/out\"\n+\tmkdir -p \"$junit_xml_dir\"\n+\tjunit_xml_base=${0##*/}\n+\tjunit_xml_path=\"$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml\"\n+\tjunit_attrs=\"name=\\\"${junit_xml_base%.sh}\\\"\"\n+\tjunit_attrs=\"$junit_attrs timestamp=\\\"$(TZ=UTC \\\n+\t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n+\twrite_junit_xml --truncate \"\" \" \"\n+\tjunit_suite_start=$(test-tool date getnanos)\n+fi\n+\n # Provide an implementation of the 'yes' utility\n yes () {\n \tif test $# = 0\n-- \ngitgitgadget\n\n"},{"id":"367434","messageId":"5d593acee5742a82c74f24f8e3f04ba3a8bc6d42.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 13/21] README: add a build badge (status of the Azure Pipelines build)","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:24Z","receivedAt":"2019-01-23T14:40:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nJust like so many other OSS projects, we now also have a build badge.\n\nSigned-off-by: Johannes Schindelin \n---\n README.md | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/README.md b/README.md\nindex f920a42fad..764c480c66 100644\n--- a/README.md\n+++ b/README.md\n@@ -1,3 +1,5 @@\n+[![Build Status](https://dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2)\n+\n Git - fast, scalable, distributed revision control system\n =========================================================\n \n-- \ngitgitgadget\n\n"},{"id":"367435","messageId":"5bdc6a08a8b8040de3082b1690f16538fcc08682.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 10/21] ci: move the Windows job to the top","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:21Z","receivedAt":"2019-01-23T14:40:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe Windows job currently takes a whopping ~1h20m to complete. Which is\n*far* longer than the next-longest job takes (linux-gcc, ~35m). As such,\nit makes sense to start the Windows job first, to minimize the overall\nrun time (which is now pretty safely the run time of the Windows job).\n\nThis affects only the Azure Pipelines configuration, not the Travis one,\nof course, as Travis cannot run our Windows job: 1h20m is distinctly\nlonger than the 50 minute timeout of Travis' free tier.\n\nThis commit is best viewed with `--color-moved`.\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 172 ++++++++++++++++++++++----------------------\n 1 file changed, 86 insertions(+), 86 deletions(-)\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex f3cabb0dd0..e44d2733a4 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -3,6 +3,92 @@ resources:\n fetchDepth: 1\n \n jobs:\n+- job: windows\n+ displayName: Windows\n+ condition: succeeded()\n+ pool: Hosted\n+ timeoutInMinutes: 240\n+ steps:\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n+ }\n+ displayName: 'Mount test-cache'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - powershell: |\n+ # Helper to check the error level of the latest command (exit with error when appropriate)\n+ function c() { if (!$?) { exit(1) } }\n+\n+ # Add build agent's MinGit to PATH\n+ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n+\n+ # Helper to initialize (or update) a Git worktree\n+ function init ($path, $url, $set_origin) {\n+ if (Test-Path $path) {\n+ cd $path; c\n+ if (Test-Path .git) {\n+ & git init; c\n+ } else {\n+ & git status\n+ }\n+ } else {\n+ & git init $path; c\n+ cd $path; c\n+ }\n+ & git config core.autocrlf false; c\n+ & git config core.untrackedCache true; c\n+ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n+ & git remote add origin $url; c\n+ }\n+ & git fetch --depth=1 $url master; c\n+ & git reset --hard FETCH_HEAD; c\n+ & git clean -df; c\n+ }\n+\n+ # Initialize Git for Windows' SDK\n+ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n+ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+\n+ # Let Git ignore the SDK and the test-cache\n+ \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n+ displayName: 'Initialize the Git for Windows SDK'\n+ - powershell: |\n+ & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ export MAKEFLAGS=-j10\n+ export DEVELOPER=1\n+ export NO_PERL=1\n+ export NO_SVN_TESTS=1\n+ export GIT_TEST_SKIP_REBASE_P=1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+ \"@\n+ if (!$?) { exit(1) }\n+ displayName: 'Build & Test'\n+ env:\n+ HOME: $(Build.SourcesDirectory)\n+ MSYSTEM: MINGW64\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'Unmount test-cache'\n+ condition: true\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'windows'\n+ platform: Windows\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+\n - job: linux_clang\n displayName: linux-clang\n condition: succeeded()\n@@ -153,92 +239,6 @@ jobs:\n publishRunAttachments: false\n condition: succeededOrFailed()\n \n-- job: windows\n- displayName: Windows\n- condition: succeeded()\n- pool: Hosted\n- timeoutInMinutes: 240\n- steps:\n- - powershell: |\n- if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n- net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n- cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n- }\n- displayName: 'Mount test-cache'\n- env:\n- GITFILESHAREPWD: $(gitfileshare.pwd)\n- - powershell: |\n- # Helper to check the error level of the latest command (exit with error when appropriate)\n- function c() { if (!$?) { exit(1) } }\n-\n- # Add build agent's MinGit to PATH\n- $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n-\n- # Helper to initialize (or update) a Git worktree\n- function init ($path, $url, $set_origin) {\n- if (Test-Path $path) {\n- cd $path; c\n- if (Test-Path .git) {\n- & git init; c\n- } else {\n- & git status\n- }\n- } else {\n- & git init $path; c\n- cd $path; c\n- }\n- & git config core.autocrlf false; c\n- & git config core.untrackedCache true; c\n- if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n- & git remote add origin $url; c\n- }\n- & git fetch --depth=1 $url master; c\n- & git reset --hard FETCH_HEAD; c\n- & git clean -df; c\n- }\n-\n- # Initialize Git for Windows' SDK\n- $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n- init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n-\n- # Let Git ignore the SDK and the test-cache\n- \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n- displayName: 'Initialize the Git for Windows SDK'\n- - powershell: |\n- & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n- export MAKEFLAGS=-j10\n- export DEVELOPER=1\n- export NO_PERL=1\n- export NO_SVN_TESTS=1\n- export GIT_TEST_SKIP_REBASE_P=1\n-\n- ci/run-build-and-tests.sh || {\n- ci/print-test-failures.sh\n- exit 1\n- }\n- \"@\n- if (!$?) { exit(1) }\n- displayName: 'Build & Test'\n- env:\n- HOME: $(Build.SourcesDirectory)\n- MSYSTEM: MINGW64\n- - powershell: |\n- if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n- cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n- }\n- displayName: 'Unmount test-cache'\n- condition: true\n- env:\n- GITFILESHAREPWD: $(gitfileshare.pwd)\n- - task: PublishTestResults@2\n- displayName: 'Publish Test Results **/TEST-*.xml'\n- inputs:\n- mergeTestResults: true\n- testRunTitle: 'windows'\n- platform: Windows\n- publishRunAttachments: false\n- condition: succeededOrFailed()\n-\n - job: linux32\n displayName: Linux32\n condition: succeeded()\n-- \ngitgitgadget\n\n"},{"id":"367436","messageId":"178dc9b789caef8e2402ab518bb5ca36da2c32ad.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 08/21] ci/lib.sh: add support for Azure Pipelines","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:20Z","receivedAt":"2019-01-23T14:40:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis patch introduces a conditional arm that defines some environment\nvariables and a function that displays the URL given the job id (to\nidentify previous runs for known-good trees).\n\nFor example, we do not have to install the git-lfs and gettext packages\non Azure Pipelines' macOS agents: they are already installed, and trying\nto install them again would result in an error.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 4456dbbcb0..a9eb4f4eae 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -20,6 +20,29 @@ then\n \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+elif test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n+then\n+\t# We are running in Azure Pipelines\n+\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n+\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n+\tCI_JOB_ID=\"$BUILD_BUILDID\"\n+\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n+\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n+\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n+\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n+\tCC=\"${CC:-gcc}\"\n+\n+\t# use a subdirectory of the cache dir (because the file share is shared\n+\t# among *all* phases)\n+\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n+\n+\turl_for_job_id () {\n+\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n fi\n \n skip_branch_tip_with_tag () {\n-- \ngitgitgadget\n\n"},{"id":"367437","messageId":"pull.31.v4.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v3.git.gitgitgadget@gmail.com","subject":"[PATCH v4 00/21] Offer to run CI/PR builds in Azure Pipelines","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:13Z","receivedAt":"2019-01-23T14:40:48Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"For a long time already, we have tested Git's source code continuously via\nTravis CI, see e.g. https://travis-ci.org/git/git/builds/421738884. It has\nserved us well, and more and more developers actually pay attention and\nbenefit from the testing this gives us.\n\nIt is also an invaluable tool for contributors who can validate their code\ncontributions via PRs on GitHub, e.g. to verify that their tests do actually\nrun on macOS (i.e. with the BSD family of Unix tools instead of the GNU\none).\n\nThe one sad part about this is the Windows support. Travis used to lack it\n(Travis announced some early support for Windows, hot off the press\n[https://blog.travis-ci.com/2018-10-11-windows-early-release]), and we work\naround that by using Azure Pipelines (the CI part of Azure DevOps, formerly\nknown as Visual Studio Team Services) indirectly: one phase in Travis would\ntrigger a build, wait for its log, and then paste that log.\n\nAs Git's Windows builds (and tests!) take quite a bit of time, Travis often\ntimed out, or somehow the trigger did not work, and for security reasons\n(the Windows builds are performed in a private pool of containers), the\nWindows builds are completely disabled for Pull Requests on GitHub.\n\nOne might ask why we did not use Azure Pipelines directly. There were a\ncouple of reasons for that:\n\n * most notably, Azure Pipelines' build logs could not be viewed\n anonymously,\n * while Azure Pipelines had Linux and Windows agents, it lacked macOS\n agents,\n * etc\n\nThe main two reasons no longer apply: macOS agents are available now\n[https://docs.microsoft.com/en-us/azure/devops/release-notes/2018/jul-10-vsts]\n, and are public projects\n[https://docs.microsoft.com/en-us/azure/devops/organizations/public/about-public-projects] \nnow, i.e. it is possible to configure a Azure Pipelines project so that \nanybody can view the logs. Since I offered v1, Azure Pipelines has been made\navailable via the GitHub Marketplace, free of cost for open source projects.\n\nI had secured such a public project for Git for Windows already, and I also\ngot one for Git. For now, the latter is hooked up with my personal git.git\nfork on GitHub, but it is my hope that I convince y'all that these Azure\nPipelines builds are a good idea, and then hook it up with \nhttps://github.com/git/git.\n\nAs a special treat, this patch series adds the ability to present the\noutcome of Git's test suite as JUnit-style .xml files. This allows the Azure\nPipelines build to present fun diagrams, trends, and makes it a lot easier\nto drill down to test failures than before. See for example \nhttps://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details\n[https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details] \n(you can click on the label of the failed test, and then see the detailed\noutput in the right pane).\n\nBut maybe you're not interested as much in the presentation of test failures\nas in the speed? Well, I got you covered with that, too. As of v3, the test\nsuite is run in parallel on Windows, cutting the overall run time to ~33\nminutes (see one of the latest builds\n[https://dev.azure.com/git/git/_build/results?buildId=302&view=logs], for\nexample).\n\nThis patch series took waaaaaaaay more time than I had originally\nanticipated, but I think that in particular the advanced display of the test\nresults and the reduction of the overall run time was worth it. Please let\nme know what you think about this.\n\nChanges since v3:\n\n * Fixed the incorrect usage of cut -c that resulted in an empty trace when\n failed tests were published (except if the very first test case in a test\n script failed, then it was correct by mistake).\n * Excluded the previous test case's \"ok\" (or \"not ok\") line from the trace\n published with failed tests.\n * Renamed TRAVIS_BRANCH to CI_BRANCH already in the first commit, as we\n should not override TRAVIS_BRANCH with something that it is not.\n * Rebased onto current master to avoid merge conflicts with the\n recently-merged sg/stress-test branch (no, Junio, I really trust myself\n more than you to resolve those merge conflicts).\n\nChanges since v2:\n\n * Removed left-over debugging code that would skip a good chunk of \n t0000-init.sh.\n * Fixed the URL of the build badge.\n * Removed a trailing empty line from, and added a missing closing pointy\n bracket to, ci/mount-fileshare.sh.\n * Moved the \"travis: fix skipping tagged releases\" commit up to the\n beginning of the patch series.\n * The commit message of \"ci/lib.sh: add support for Azure Pipelines\" now\n mentions explicitly that the Homebrew packages that need to be installed\n on Travis' macOS agents are already installed on Azure Pipelines'.\n * Some commands were not guarded by || exit 1, i.e. if they would fail, the\n build would not have failed.\n * We now install gcc-8 for the linux-gcc job.\n * We no longer try to re-run failed tests with verbose log. Instead, we\n simply use the verbose log to begin with. Tests showed that it had a\n negligible impact on the overall run time.\n * The test_atexit_handler function was scratched; It would be the right\n thing to do, but is actually an independent topic (it was only\n implemented in v2 to accommodate the \"re-run with verbose log on failure\"\n trick)\n * We now use a new YAML schema (most notably, \"phase\" is now known as\n \"job\")\n * The Windows job contained PowerShell sections that were indented with 3\n spaces instead of 2.\n * The Windows job is now structured better, by separating different\n concerns into different \"tasks\" so that it is easier to see what exactly\n failed (was it the build? was it the test?)\n * The Windows job was split into a job to build Git and 10 parallel jobs to\n run the test suite with the artifacts built by the first job. This\n reduces the overall run time from ~1h20 (which was the run time by the\n Windows job) to ~35 minutes (which is the run time of the linux-gcc job).\n * The JUnit XML is now written using a test helper rather than a fragile\n and probably not even portable sed call.\n * Since we needed to determine the file size of the verbose log (to cut out\n individual test cases' log output), we now introduce a test helper to do\n that, and use it throughout the test suite (where Perl was used before).\n * It would appear that a recent change either in Cygwin or in the Azure VMs\n causes problems sporadically where the trash directories cannot be\n removed, but a subsequent rm will succeed. We now simply do that, because\n it won't harm the common case (where the first rm succeeds already) and\n because it helps the Windows job succeed pretty reliably.\n\nChanges since v1:\n\n * Removed a superfluous eval.\n * Added the commit that fixes the Travis PR builds targeting master that \n just happens to be tagged (see \n https://travis-ci.org/git/git/jobs/424276413 for an incorrectly-skipped\n build).\n * The commit messages and the cover letter now reflect the name change from\n Visual Studio Team Services to Azure DevOps (and in particular, Azure\n Pipelines for the automated builds).\n * Now we're using test_atexit (which we introduced for that purpose)\n instead of hard-coding kill_p4d and stop_git_daemon.\n * The build should now also succeed for Pull Requests (where secret\n variables are not available, for security reasons, and as a consequence\n the file share cannot be mounted).\n * The shell scripted parts now use proper && chains.\n\nJohannes Schindelin (21):\n travis: fix skipping tagged releases\n ci: rename the library of common functions\n ci/lib.sh: encapsulate Travis-specific things\n ci: inherit --jobs via MAKEFLAGS in run-build-and-tests\n ci: use a junction on Windows instead of a symlink\n test-date: add a subcommand to measure times in shell scripts\n tests: optionally write results as JUnit-style .xml\n ci/lib.sh: add support for Azure Pipelines\n Add a build definition for Azure DevOps\n ci: move the Windows job to the top\n ci: use git-sdk-64-minimal build artifact\n mingw: be more generous when wrapping up the setitimer() emulation\n README: add a build badge (status of the Azure Pipelines build)\n tests: avoid calling Perl just to determine file sizes\n tests: include detailed trace logs with --write-junit-xml upon failure\n mingw: try to work around issues with the test cleanup\n tests: add t/helper/ to the PATH with --with-dashes\n t0061: fix with --with-dashes and RUNTIME_PREFIX\n tests: optionally skip bin-wrappers/\n ci: speed up Windows phase\n ci: parallelize testing on Windows\n\n Makefile | 11 +\n README.md | 2 +\n azure-pipelines.yml | 344 ++++++++++++++++++++++++++++\n ci/install-dependencies.sh | 5 +-\n ci/{lib-travisci.sh => lib.sh} | 85 +++++--\n ci/make-test-artifacts.sh | 12 +\n ci/mount-fileshare.sh | 25 ++\n ci/print-test-failures.sh | 4 +-\n ci/run-build-and-tests.sh | 9 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/run-test-slice.sh | 17 ++\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 3 +-\n compat/mingw.c | 2 +-\n t/.gitignore | 1 +\n t/README | 9 +\n t/helper/test-date.c | 12 +\n t/helper/test-path-utils.c | 64 ++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/helper/test-xml-encode.c | 80 +++++++\n t/t0021-conversion.sh | 2 +-\n t/t0061-run-command.sh | 3 +-\n t/t1050-large.sh | 2 +-\n t/t5315-pack-objects-compression.sh | 2 +-\n t/t9303-fast-import-compression.sh | 2 +-\n t/test-lib.sh | 138 ++++++++++-\n 28 files changed, 796 insertions(+), 46 deletions(-)\n create mode 100644 azure-pipelines.yml\n rename ci/{lib-travisci.sh => lib.sh} (52%)\n create mode 100755 ci/make-test-artifacts.sh\n create mode 100755 ci/mount-fileshare.sh\n create mode 100755 ci/run-test-slice.sh\n create mode 100644 t/helper/test-xml-encode.c\n\n\nbase-commit: 16a465bc018d09e9d7bbbdc5f40a7fb99c21f8ef\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-31%2Fdscho%2Fvsts-ci-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-31/dscho/vsts-ci-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/31\n\nRange-diff vs v3:\n\n 1: 75ec97b392 < -: ---------- travis: fix skipping tagged releases\n -: ---------- > 1: 6ed2f2a35c travis: fix skipping tagged releases\n 2: d520f45108 = 2: b086f36660 ci: rename the library of common functions\n 3: 06fa564386 ! 3: 2d0b62f186 ci/lib.sh: encapsulate Travis-specific things\n @@ -30,7 +30,7 @@\n \n -# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n -# want here. We want the source branch instead.\n --TRAVIS_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n +-CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n +if test true = \"$TRAVIS\"\n +then\n +\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n @@ -55,20 +55,6 @@\n \n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n -@@\n - \t# we can skip the build because we won't be skipping a build\n - \t# of a tag.\n - \n --\tif TAG=$(git describe --exact-match \"$TRAVIS_BRANCH\" 2>/dev/null) &&\n --\t\ttest \"$TAG\" != \"$TRAVIS_BRANCH\"\n -+\tif TAG=$(git describe --exact-match \"$CI_BRANCH\" 2>/dev/null) &&\n -+\t\ttest \"$TAG\" != \"$CI_BRANCH\"\n - \tthen\n --\t\techo \"$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)\"\n -+\t\techo \"$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)\"\n - \t\texit 0\n - \tfi\n - }\n @@\n # job if we encounter the same tree again and can provide a useful info\n # message.\n 4: 52fb8e72fb = 4: 83b92a87e7 ci: inherit --jobs via MAKEFLAGS in run-build-and-tests\n 5: a35bc43a04 = 5: 6c9eb4e33e ci: use a junction on Windows instead of a symlink\n 6: 16090ff67c = 6: aa053ed993 test-date: add a subcommand to measure times in shell scripts\n 7: 272c0c0446 ! 7: 07fb37e0a6 tests: optionally write results as JUnit-style .xml\n @@ -148,15 +148,15 @@\n --- a/t/test-lib.sh\n +++ b/t/test-lib.sh\n @@\n - \t-V|--verbose-log)\n \t\tverbose_log=t\n - \t\tshift ;;\n + \t\ttee=t\n + \t\t;;\n +\t--write-junit-xml)\n +\t\twrite_junit_xml=t\n -+\t\tshift ;;\n - \t*)\n - \t\techo \"error: unknown test option '$1'\" >&2; exit 1 ;;\n - \tesac\n ++\t\t;;\n + \t--stress)\n + \t\tstress=t ;;\n + \t--stress=*)\n @@\n # the test_expect_* functions instead.\n \n @@ -278,7 +278,7 @@\n +\n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n - \t\ttest_results_dir=\"$TEST_OUTPUT_DIRECTORY/test-results\"\n + \t\tmkdir -p \"$TEST_RESULTS_DIR\"\n @@\n else\n \tmkdir -p \"$TRASH_DIRECTORY\"\n 8: fefab79d46 = 8: 178dc9b789 ci/lib.sh: add support for Azure Pipelines\n 9: ddee8ecd8a = 9: eaba471b89 Add a build definition for Azure DevOps\n 10: 58bb8d0469 = 10: 5bdc6a08a8 ci: move the Windows job to the top\n 11: b6bb6a87a3 = 11: 93eebb74ce ci: use git-sdk-64-minimal build artifact\n 12: 2b6ee78f02 = 12: 18d76823e5 mingw: be more generous when wrapping up the setitimer() emulation\n 13: df326039d9 = 13: 5d593acee5 README: add a build badge (status of the Azure Pipelines build)\n 14: 0d547db8f7 = 14: 3083041e33 tests: avoid calling Perl just to determine file sizes\n 15: f678b105f8 < -: ---------- tests: include detailed trace logs with --write-junit-xml upon failure\n -: ---------- > 15: 1c44d3b8b8 tests: include detailed trace logs with --write-junit-xml upon failure\n 16: 8ef674a236 = 16: 59c1194ae2 mingw: try to work around issues with the test cleanup\n 17: b503167084 = 17: 966c412f03 tests: add t/helper/ to the PATH with --with-dashes\n 18: 713910e1dc = 18: d613c79aff t0061: fix with --with-dashes and RUNTIME_PREFIX\n 19: 3a77eafb44 ! 19: a198885b76 tests: optionally skip bin-wrappers/\n @@ -31,13 +31,13 @@\n --- a/t/test-lib.sh\n +++ b/t/test-lib.sh\n @@\n - \t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t; shift ;;\n + \t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n \t--with-dashes)\n - \t\twith_dashes=t; shift ;;\n + \t\twith_dashes=t ;;\n +\t--no-bin-wrappers)\n -+\t\tno_bin_wrappers=t; shift ;;\n ++\t\tno_bin_wrappers=t ;;\n \t--no-color)\n - \t\tcolor=; shift ;;\n + \t\tcolor= ;;\n \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n @@\n \tPATH=$GIT_TEST_INSTALLED:$GIT_BUILD_DIR/t/helper:$PATH\n 20: fba97133bf = 20: 6520f4603c ci: speed up Windows phase\n 21: e568349930 ! 21: 8bdd9804a1 ci: parallelize testing on Windows\n @@ -229,7 +229,7 @@ $^\n {\n \tif (argc == 3 && !strcmp(argv[1], \"normalize_path_copy\")) {\n @@\n - \t\treturn !!res;\n + \t\treturn 0;\n \t}\n \n +\tif (argc > 5 && !strcmp(argv[1], \"slice-tests\")) {\n\n-- \ngitgitgadget\n"},{"id":"367438","messageId":"6ed2f2a35c394987be2bebd2364915c4daed888d.1548254412.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v4 01/21] travis: fix skipping tagged releases","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-23T14:40:14Z","receivedAt":"2019-01-23T14:40:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWhen building a PR, TRAVIS_BRANCH refers to the *target branch*.\nTherefore, if a PR targets `master`, and `master` happened to be tagged,\nwe skipped the build by mistake.\n\nFix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*)\nwhen available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also\nknown as \"push builds\").\n\nLet's give it a new variable name, too: CI_BRANCH (as it is different\nfrom TRAVIS_BRANCH). This also prepares for the upcoming patches which\nwill make our ci/* code a bit more independent from Travis and open it\nto other CI systems (in particular to Azure Pipelines).\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib-travisci.sh | 25 ++++++++++++++-----------\n 1 file changed, 14 insertions(+), 11 deletions(-)\n\ndiff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh\nindex 69dff4d1ec..c26bb6a274 100755\n--- a/ci/lib-travisci.sh\n+++ b/ci/lib-travisci.sh\n@@ -1,22 +1,25 @@\n # Library of functions shared by all CI scripts\n \n+# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n+# want here. We want the source branch instead.\n+CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+\n skip_branch_tip_with_tag () {\n \t# Sometimes, a branch is pushed at the same time the tag that points\n \t# at the same commit as the tip of the branch is pushed, and building\n \t# both at the same time is a waste.\n \t#\n-\t# Travis gives a tagname e.g. v2.14.0 in $TRAVIS_BRANCH when\n-\t# the build is triggered by a push to a tag. Let's see if\n-\t# $TRAVIS_BRANCH is exactly at a tag, and if so, if it is\n-\t# different from $TRAVIS_BRANCH. That way, we can tell if\n-\t# we are building the tip of a branch that is tagged and\n-\t# we can skip the build because we won't be skipping a build\n-\t# of a tag.\n-\n-\tif TAG=$(git describe --exact-match \"$TRAVIS_BRANCH\" 2>/dev/null) &&\n-\t\ttest \"$TAG\" != \"$TRAVIS_BRANCH\"\n+\t# When the build is triggered by a push to a tag, $CI_BRANCH will\n+\t# have that tagname, e.g. v2.14.0. Let's see if $CI_BRANCH is\n+\t# exactly at a tag, and if so, if it is different from $CI_BRANCH.\n+\t# That way, we can tell if we are building the tip of a branch that\n+\t# is tagged and we can skip the build because we won't be skipping a\n+\t# build of a tag.\n+\n+\tif TAG=$(git describe --exact-match \"$CI_BRANCH\" 2>/dev/null) &&\n+\t\ttest \"$TAG\" != \"$CI_BRANCH\"\n \tthen\n-\t\techo \"$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)\"\n+\t\techo \"$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)\"\n \t\texit 0\n \tfi\n }\n-- \ngitgitgadget\n\n"},{"id":"367441","messageId":"nycvar.QRO.7.76.6.1901231713200.41@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"Fast CI for all branches in gitster/git, was Re: [PATCH v4 00/21] Offer to run CI/PR builds in Azure Pipelines","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-23T16:23:34Z","receivedAt":"2019-01-23T16:23:57Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi all,\n\nOn Wed, 23 Jan 2019, Johannes Schindelin via GitGitGadget wrote:\n\n> * The Windows job was split into a job to build Git and 10 parallel\n> jobs to run the test suite with the artifacts built by the first job.\n> This reduces the overall run time from ~1h20 (which was the run time by\n> the Windows job) to ~35 minutes (which is the run time of the linux-gcc\n> job).\n\nThis improvement in run time is something I was very, very excited about,\nbecause it finally lets me use the cloud as my build & test machine,\neffectively.\n\nAnd not only me! You can use it, too, because I ported part of this Azure\nPipeline to GitGitGadget's Git fork, it is now the default PR build. It\ntakes about 20 minutes to complete if nothing else is building at the\ntime, and if it is all green, you can be relatively sure that your code is\ngood on Windows, macOS and Linux.\n\nIf you want to test your code that way, just open a PR on\nhttps://github.com/gititgadget/git and wait for what GitHub calls \"Check\"\nto complete.\n\nSince the Pipeline is now so fast, and since I have to mirror the branches\nin gitster/git to gitgitgadget/git anyway (so that you can contribute\npatch series on top of other patch series that are in-flight), I also\ndecided to turn on the same PR build as CI build: whenever Junio pushes a\nnew branch or a branch update (and when that change is picked up by the\nPipeline that mirrors it into gitgitgadget/git), the Pipeline will run, so\nI will never have to bisect `pu` again (which is a pain due to the many,\nmany, many merge bases) but instead the respective broken branch (because\nI can now see which branch is broken directly).\n\nYou can even see those build outcomes yourself at\n\n\thttps://github.com/gitgitgadget/git/branches/active\n\nEach of those checkmarks (or red X marks) represents a build, and you can\nclick on it to see the Checks, click on the Details of \"CI for\nGitGitGadget's Git fork\" and finally follow the \"View more details on\nAzure Pipelines\" link to see the full build log.\n\nFor example, the build log of the ph/pack-objects-mutex-fix branch is\nhere: https://dev.azure.com/gitgitgadget/git/_build/results?buildId=248\n\nEnjoy,\nJohannes\n"},{"id":"367495","messageId":"xmqqpnsn7zgr.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"5bdc6a08a8b8040de3082b1690f16538fcc08682.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 10/21] ci: move the Windows job to the top","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T22:59:00Z","receivedAt":"2019-01-23T22:59:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> The Windows job currently takes a whopping ~1h20m to complete. Which is\n> *far* longer than the next-longest job takes (linux-gcc, ~35m). As such,\n> it makes sense to start the Windows job first, to minimize the overall\n> run time (which is now pretty safely the run time of the Windows job).\n\nIs the reason why Windows job gets started first is to make sure\nthat it, which is known to take the longest time, never has to wait\nbefore starting while other jobs run, in case there is limited\nparallelism? The last part of this sentence is what readers of this\nstep will need in order to be convinced by the justification given,\nbecause (1) if the jobs run totally serially, the order does not\nmatter much---if anything, running shorter jobs first would give\nresults from more jobs sooner, and (2) if the jobs run totally in\nparallel, the order does not matter as long as we have enough\nparallelism.\n\n> This commit is best viewed with `--color-moved`.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> azure-pipelines.yml | 172 ++++++++++++++++++++++----------------------\n> 1 file changed, 86 insertions(+), 86 deletions(-)\n\nFor those who are seeing this azure-pipelines series for the first\ntime, it would probably be unclear what the point of adding an\nentire file in 09/21 and them moving lines around in 10/21 is. If\nsomebody asked me why, I wouldn't be able to explain why it is a\ngood idea.\n\nWould it hurt readability if these two steps are combined?\n\n"},{"id":"367497","messageId":"xmqqbm476j1i.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"6ed2f2a35c394987be2bebd2364915c4daed888d.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 01/21] travis: fix skipping tagged releases","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T22:00:49Z","receivedAt":"2019-01-23T23:39:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> When building a PR, TRAVIS_BRANCH refers to the *target branch*.\n> Therefore, if a PR targets `master`, and `master` happened to be tagged,\n> we skipped the build by mistake.\n>\n> Fix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*)\n> when available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also\n> known as \"push builds\").\n>\n> Let's give it a new variable name, too: CI_BRANCH (as it is different\n> from TRAVIS_BRANCH). This also prepares for the upcoming patches which\n> will make our ci/* code a bit more independent from Travis and open it\n> to other CI systems (in particular to Azure Pipelines).\n\nMakes sense.\n"},{"id":"367498","messageId":"xmqq5zuf6j1h.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"b086f366606d97097007578ba2ad1523827a4dd7.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 02/21] ci: rename the library of common functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T22:01:55Z","receivedAt":"2019-01-23T23:39:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> The name is hard-coded to reflect that we use Travis CI for continuous\n> testing.\n>\n> In the next commits, we will extend this to be able use Azure DevOps,\n> too.\n>\n> So let's adjust the name to make it more generic.\n\nYup. If we do not like lib-travis, it is possible to call it\nlib-ci, but because these are all in ci/ directory already, there is\nno reason to be extra redundant.\n\nMakes sense.\n"},{"id":"367499","messageId":"xmqqimyf54gx.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"178dc9b789caef8e2402ab518bb5ca36da2c32ad.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 08/21] ci/lib.sh: add support for Azure Pipelines","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T22:40:08Z","receivedAt":"2019-01-23T23:39:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> This patch introduces a conditional arm that defines some environment\n> variables and a function that displays the URL given the job id (to\n> identify previous runs for known-good trees).\n>\n> For example, we do not have to install the git-lfs and gettext packages\n> on Azure Pipelines' macOS agents: they are already installed, and trying\n> to install them again would result in an error.\n\nThe rationale in the second paragraph is shared with the earlier one\nthat introduced BREW_INSTALL_PACKAGES variable in the first place.\nIt would have made more sense if it was explained there (i.e. say\n\"in environments other than Travis, some packages may be pre\ninstalled and asking to install them again may cause failure, so\nmake it configurable what to install\")---that way, the explanation\nin this step's log message can just say \"Because Azure pipeline's\nmacOS agents already have git-lfs and gettext installed, unlike\nTravis, we can leave BREW_INSTALL_PACKAGES empty\".\n\n> Signed-off-by: Johannes Schindelin \n> ---\n> ci/lib.sh | 23 +++++++++++++++++++++++\n> 1 file changed, 23 insertions(+)\n>\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 4456dbbcb0..a9eb4f4eae 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -20,6 +20,29 @@ then\n> \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n> \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n> \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n> +elif test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n> +then\n> +\t# We are running in Azure Pipelines\n> +\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n> +\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n> +\tCI_JOB_ID=\"$BUILD_BUILDID\"\n> +\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n> +\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n> +\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n> +\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n> +\tCC=\"${CC:-gcc}\"\n> +\n> +\t# use a subdirectory of the cache dir (because the file share is shared\n> +\t# among *all* phases)\n> +\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n> +\n> +\turl_for_job_id () {\n> +\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n> +\t}\n> +\n> +\tBREW_INSTALL_PACKAGES=\n> +\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> +\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n> fi\n> \n> skip_branch_tip_with_tag () {\n"},{"id":"367500","messageId":"xmqqo98754gy.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"aa053ed9936ebae0ca5e18d27de96f1d054d7f89.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 06/21] test-date: add a subcommand to measure times in shell scripts","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T22:29:02Z","receivedAt":"2019-01-23T23:39:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> In the next commit, we want to teach Git's test suite to optionally\n> output test results in JUnit-style .xml files. These files contain\n> information about the time spent. So we need a way to measure time.\n>\n> While we could use `date +%s` for that, this will give us only seconds,\n> i.e. very coarse-grained timings.\n>\n> GNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output),\n> but there is no equivalent in BSD `date` (read: on macOS, we would not\n> be able to obtain precise timings).\n>\n> So let's introduce `test-tool date getnanos`, with an optional start\n> time, that outputs preciser values.\n\nI think the goal to have our own stopwatch so that we do not have to\nworry about differences among system-provided ones makes sense.\n\nThe only thing that may become an issue is how widely available\ngetnanotime() is. As \"test-date\" itself is built on any platform an\nend-user/developer runs our tests, which is wider set of platforms\nthan what we run Travis and other CIs on, unconditionally relying on\nits availability might pose an issue. I dunno.\n\n> diff --git a/t/helper/test-date.c b/t/helper/test-date.c\n> index a0837371ab..792a805374 100644\n> --- a/t/helper/test-date.c\n> +++ b/t/helper/test-date.c\n> @@ -7,6 +7,7 @@ static const char *usage_msg = \"\\n\"\n> \" test-tool date parse [date]...\\n\"\n> \" test-tool date approxidate [date]...\\n\"\n> \" test-tool date timestamp [date]...\\n\"\n> +\" test-tool date getnanos [start-nanos]\\n\"\n> \" test-tool date is64bit\\n\"\n> \" test-tool date time_t-is64bit\\n\";\n> \n> @@ -82,6 +83,15 @@ static void parse_approx_timestamp(const char **argv, struct timeval *now)\n> \t}\n> }\n> \n> +static void getnanos(const char **argv, struct timeval *now)\n> +{\n> +\tdouble seconds = getnanotime() / 1.0e9;\n> +\n> +\tif (*argv)\n> +\t\tseconds -= strtod(*argv, NULL);\n> +\tprintf(\"%lf\\n\", seconds);\n> +}\n> +\n> int cmd__date(int argc, const char **argv)\n> {\n> \tstruct timeval now;\n> @@ -108,6 +118,8 @@ int cmd__date(int argc, const char **argv)\n> \t\tparse_approxidate(argv+1, &now);\n> \telse if (!strcmp(*argv, \"timestamp\"))\n> \t\tparse_approx_timestamp(argv+1, &now);\n> +\telse if (!strcmp(*argv, \"getnanos\"))\n> +\t\tgetnanos(argv+1, &now);\n> \telse if (!strcmp(*argv, \"is64bit\"))\n> \t\treturn sizeof(timestamp_t) == 8 ? 0 : 1;\n> \telse if (!strcmp(*argv, \"time_t-is64bit\"))\n"},{"id":"367501","messageId":"xmqq7eev54gv.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"5bdc6a08a8b8040de3082b1690f16538fcc08682.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 10/21] ci: move the Windows job to the top","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T23:07:17Z","receivedAt":"2019-01-23T23:39:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> The Windows job currently takes a whopping ~1h20m to complete. Which is\n> *far* longer than the next-longest job takes (linux-gcc, ~35m). As such,\n> it makes sense to start the Windows job first, to minimize the overall\n> run time (which is now pretty safely the run time of the Windows job).\n\nIs the reason why Windows job gets started first is to make sure\nthat it, which is known to take the longest time, never has to wait\nbefore starting while other jobs run, in case there is limited\nparallelism? The last part of this sentence is what readers of this\nstep will need in order to be convinced by the justification given,\nbecause (1) if the jobs run totally serially, the order does not\nmatter much---if anything, running shorter jobs first would give\nresults from more jobs sooner, and (2) if the jobs run totally in\nparallel, the order does not matter as long as we have enough\nparallelism.\n\n> This commit is best viewed with `--color-moved`.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> azure-pipelines.yml | 172 ++++++++++++++++++++++----------------------\n> 1 file changed, 86 insertions(+), 86 deletions(-)\n\nFor those who are seeing this azure-pipelines series for the first\ntime, it would probably be unclear what the point of adding an\nentire file in 09/21 and them moving lines around in 10/21 is. If\nsomebody asked me why, I wouldn't be able to explain why it is a\ngood idea.\n\nThe same comment applies to 11/21.\n\nWould it hurt readability if these steps are combined?\n\nIf 09/21 were \"copy travis.yml to create a moral-equivalent set-up\nfor azure.yml\", then it is an entirely different story (i.e. \"we\nstart from an equivalent setup as we have, and then tweak to match\nour needs better, and we can view the tweak easier as a separate\nstep\"), but I did not get the impression that it was what happened\nthere in 09/21.\n"},{"id":"367503","messageId":"xmqqtvhz54gz.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"83b92a87e7698cee1e2c44252b934ad436d75a90.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 04/21] ci: inherit --jobs via MAKEFLAGS in run-build-and-tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T22:22:10Z","receivedAt":"2019-01-23T23:39:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> Let's not decide in the generic ci/ script how many jobs to run in\n> parallel; it is easy enough to hand that information down via the\n> `MAKEFLAGS`.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> ci/run-build-and-tests.sh | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n>\n> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\n> index db342bb6a8..80d72d120f 100755\n> --- a/ci/run-build-and-tests.sh\n> +++ b/ci/run-build-and-tests.sh\n> @@ -7,7 +7,7 @@\n> \n> ln -s \"$cache_dir/.prove\" t/.prove\n> \n> -make --jobs=2\n> +make\n> make --quiet test\n> if test \"$jobname\" = \"linux-gcc\"\n> then\n\nAs there is no assignment to MAKEFLAGS in this patch, is it intended\nfor this step to change behaviour (possibly with the intention to\nadd \"default 2 jobs at least under travis\" back later in the\nseries)? Not that it matters too much, but it is unnerving to see\nthat the proposed log message promising \"it is easy enough\" while\nnot actually doing so, without expressing an intention.\n\n\n"},{"id":"367502","messageId":"xmqqd0on54gw.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"eaba471b89976b51ea19fc6772890304135ab539.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 09/21] Add a build definition for Azure DevOps","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T22:44:14Z","receivedAt":"2019-01-23T23:39:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> This commit adds an azure-pipelines.yml file which is Azure DevOps'\n> equivalent to Travis CI's .travis.yml.\n>\n> To make things a bit easier to understand, we refrain from using the\n> `matrix` feature here because (while it is powerful) it can be a bit\n> confusing to users who are not familiar with CI setups. Therefore, we\n> use a separate phase even for similar configurations (such as GCC vs\n> Clang on Linux, GCC vs Clang on macOS).\n>\n> Also, we make use of the shiny new feature we just introduced where the\n> test suite can output JUnit-style .xml files. This information is made\n> available in a nice UI that allows the viewer to filter by phase and/or\n> test number, and to see trends such as: number of (failing) tests, time\n> spent running the test suite, etc.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> azure-pipelines.yml | 306 ++++++++++++++++++++++++++++++++++++++++++\n\nWay-overlong line in this file bothers me somewhat, but let's say\nthese are not for human consumption and it is OK ;-)\n\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index a9eb4f4eae..91cf1402bf 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -42,7 +42,7 @@ then\n> \n> \tBREW_INSTALL_PACKAGES=\n> \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> -\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n> +\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n> fi\n\nInteresting to see this change here, not in the previous step. If\nwe stopped at 08/21, did we even have a chance to enter this elif\nblock? If not, it probably is sane to squash this hunk to the\nprevious one that introduced the elif block.\n\n"},{"id":"367504","messageId":"xmqqzhrr54h0.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"2d0b62f1862d5fce3ac0d7b813399f560600917d.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T22:19:16Z","receivedAt":"2019-01-23T23:39:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> index fe65144152..bcdcc71592 100755\n> --- a/ci/install-dependencies.sh\n> +++ b/ci/install-dependencies.sh\n> @@ -37,7 +37,8 @@ osx-clang|osx-gcc)\n> \tbrew update --quiet\n> \t# Uncomment this if you want to run perf tests:\n> \t# brew install gnu-time\n\nIsn't this comment now stale?\n\nIt would be, under this new arrangement of the code, most natural\nfor Those who want to use gnu-time to arrange it to be somehow added\nto $BREW_INSTALL_PACKAGES, no?\n\n> -\tbrew install git-lfs gettext\n> +\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n> +\tbrew install $BREW_INSTALL_PACKAGES\n\nIt is unclear how this relates to \"encapsulate Travis-specific\"; it\nis guessable that perhaps only under Travis we'd use\nBREW_INSTALL_PACKAGES, but it still is unexplained why under other\nCIs we don't do lfs or gettext (if that is the plan---that is not\nclear, either).\n\n> \tbrew link --force gettext\n> \tbrew install caskroom/cask/perforce\n> \t;;\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index c26bb6a274..4456dbbcb0 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -1,8 +1,26 @@\n> # Library of functions shared by all CI scripts\n> \n> -# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n> -# want here. We want the source branch instead.\n> -CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> +if test true = \"$TRAVIS\"\n> +then\n> +\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n> +\t# what we want here. We want the source branch instead.\n> +\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> +\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n> +\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n> +\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n> +\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n> +\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n> +\n> +\tcache_dir=\"$HOME/travis-cache\"\n> +\n> +\turl_for_job_id () {\n> +\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n> +\t}\n> +\n> +\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n> +\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n> +\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n> +fi\n\nOK. I actually was expecting this to be something like\n\n\tcase \"$CI_TYPE\" in\n\tTravis) ... the reindented travis specific stuff here ... ;;\n\t*) echo >&2 \"unknown CI_TYPE: $CI_TYPE\"; exit 1 ;;\n\tesac\n\nto make it equally easy to add not just the second one but the third\none.\n\n> @@ -28,7 +46,7 @@ skip_branch_tip_with_tag () {\n> # job if we encounter the same tree again and can provide a useful info\n> # message.\n> save_good_tree () {\n> -\techo \"$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID\" >>\"$good_trees_file\"\n> +\techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n\nMakes sense. $CI_COMMIT is still coming from $TRAVIS_COMMIT but is\nthat different from \"git rev-parse $CI_BRANCH\" I wonder (thinking\naloud, not suggesting any changes).\n\n> @@ -38,7 +56,7 @@ save_good_tree () {\n> ...\n> diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\n> index 7aef39a2fd..d2045b63a6 100755\n> --- a/ci/print-test-failures.sh\n> +++ b/ci/print-test-failures.sh\n> @@ -69,7 +69,7 @@ do\n> \tfi\n> done\n> \n> -if [ $combined_trash_size -gt 0 ]\n> +if [ -n \"$TRAVIS_JOB_ID\" -a $combined_trash_size -gt 0 ]\n\nI do not quite understand this change.\n\nIf this wants to say that \"we know how to show failures only when we\nknow the job id\", then testing for $CI_JOB_ID would be more\nappropriate, and if this wants to say that \"we know how to show\nfailures only when we are running under travis\", then it would be\nmore appropriate to switch on test \"$TRAVIS\" = true that is used at\nthe beginning of ci/lib.sh (or if you would switch to \"case $CI_TYPE\"\nthere, use the same construct). I cannot quite make an argument to\nsupport the use of $TRAVIS_JOB_ID in either case.\n\n> +test -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n> gem install asciidoctor\n> \n> make check-builtins\n\nOK, BREW_INSTALL_PACKAGES has an assignment in the travis specific\nsection to trigger install of two separate packages, and there is no\nassignment to ALREADY_HAVE_ASCIIDOCTOR in the travis specific\ncodepath, so we will end up installing asciidoctor here. Makes\nsense, even though this one too is under-explained just like the\nother one is.\n\n"},{"id":"367505","messageId":"xmqq1s5354gu.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"966c412f03013902826ffd188ee9af973be60833.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 17/21] tests: add t/helper/ to the PATH with --with-dashes","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T23:33:00Z","receivedAt":"2019-01-23T23:39:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> We really need to be able to find the test helpers... Really. This\n> change was forgotten when we moved the test helpers into t/helper/\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> t/test-lib.sh | 2 +-\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n\nIt is unfortunate that --with-dashes is needed for valgrind and\nGIT_TEST_INSTALLED. Otherwise I do not see why we care about\nkeeping the option working.\n\nDoes any or your test matrix entry actually use --with-dashes and\nrely on it working, or is it just the stuff that implicitly depend\non it?\n\nThanks.\n\n>\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index 9c0ca5effb..c790e98fd2 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -1227,7 +1227,7 @@ else # normal case, use ../bin-wrappers only unless $with_dashes:\n> \tGIT_EXEC_PATH=$GIT_BUILD_DIR\n> \tif test -n \"$with_dashes\"\n> \tthen\n> -\t\tPATH=\"$GIT_BUILD_DIR:$PATH\"\n> +\t\tPATH=\"$GIT_BUILD_DIR:$GIT_BUILD_DIR/t/helper:$PATH\"\n> \tfi\n> fi\n> GIT_TEMPLATE_DIR=\"$GIT_BUILD_DIR\"/templates/blt\n"},{"id":"367506","messageId":"xmqqva2f3pwd.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"6520f4603c78d38f5168164218dc108833a5583d.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 20/21] ci: speed up Windows phase","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T23:35:00Z","receivedAt":"2019-01-23T23:39:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> As Unix shell scripting comes at a hefty price on Windows, we have to\n> see where we can save some time to run the test suite.\n>\n> Let's skip the chain linting and the bin-wrappers/ redirection on\n> Windows; this seems to shave of anywhere between 10-30% from the overall\n> runtime.\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> ci/lib.sh | 2 ++\n> 1 file changed, 2 insertions(+)\n>\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 91cf1402bf..e1d5222ceb 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -43,6 +43,8 @@ then\n> \tBREW_INSTALL_PACKAGES=\n> \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> \texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n> +\ttest windows_nt != \"$CI_OS_NAME\" ||\n> +\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n\nA clever and clean workaround. Looks good.\n"},{"id":"367507","messageId":"xmqqo9873pvw.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 00/21] Offer to run CI/PR builds in Azure Pipelines","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-23T23:39:31Z","receivedAt":"2019-01-23T23:39:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> This patch series took waaaaaaaay more time than I had originally\n> anticipated, but I think that in particular the advanced display of the test\n> results and the reduction of the overall run time was worth it. Please let\n> me know what you think about this.\n\nI left comments here and there on individual steps, but overall it\nwas a pleasant read.\n\nThanks.\n"},{"id":"367657","messageId":"20190125132545.GD6702@szeder.dev","threadId":"49270","inReplyTo":"xmqqtvhz54gz.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 04/21] ci: inherit --jobs via MAKEFLAGS in run-build-and-tests","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-25T13:25:45Z","receivedAt":"2019-01-25T13:25:51Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jan 23, 2019 at 02:22:10PM -0800, Junio C Hamano wrote:\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > From: Johannes Schindelin \n> >\n> > Let's not decide in the generic ci/ script how many jobs to run in\n> > parallel; it is easy enough to hand that information down via the\n> > `MAKEFLAGS`.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > ci/run-build-and-tests.sh | 2 +-\n> > 1 file changed, 1 insertion(+), 1 deletion(-)\n> >\n> > diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\n> > index db342bb6a8..80d72d120f 100755\n> > --- a/ci/run-build-and-tests.sh\n> > +++ b/ci/run-build-and-tests.sh\n> > @@ -7,7 +7,7 @@\n> > \n> > ln -s \"$cache_dir/.prove\" t/.prove\n> > \n> > -make --jobs=2\n> > +make\n> > make --quiet test\n> > if test \"$jobname\" = \"linux-gcc\"\n> > then\n> \n> As there is no assignment to MAKEFLAGS in this patch, is it intended\n> for this step to change behaviour (possibly with the intention to\n> add \"default 2 jobs at least under travis\" back later in the\n> series)? Not that it matters too much, but it is unnerving to see\n> that the proposed log message promising \"it is easy enough\" while\n> not actually doing so, without expressing an intention.\n\nFurthermore, there are several other 'ci/run-.sh' scripts\nthat still run 'make -j N'.\n\n"},{"id":"367658","messageId":"20190125135127.GE6702@szeder.dev","threadId":"49270","inReplyTo":"2d0b62f1862d5fce3ac0d7b813399f560600917d.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-01-25T13:51:27Z","receivedAt":"2019-01-25T13:51:34Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jan 23, 2019 at 06:40:16AM -0800, Johannes Schindelin via GitGitGadget wrote:\n> From: Johannes Schindelin \n> \n> The upcoming patches will allow building git.git via Azure Pipelines\n> (i.e. Azure DevOps' Continuous Integration), where variable names and\n> URLs look a bit different than in Travis CI.\n> \n> Signed-off-by: Johannes Schindelin \n> ---\n\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index c26bb6a274..4456dbbcb0 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -1,8 +1,26 @@\n> # Library of functions shared by all CI scripts\n> \n> -# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n> -# want here. We want the source branch instead.\n> -CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> +if test true = \"$TRAVIS\"\n> +then\n> +\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n> +\t# what we want here. We want the source branch instead.\n> +\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> +\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n> +\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n> +\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n> +\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n> +\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n> +\n> +\tcache_dir=\"$HOME/travis-cache\"\n> +\n> +\turl_for_job_id () {\n> +\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n> +\t}\n> +\n> +\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n> +\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n> +\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n> +fi\n\nPlease set these variables after 'set -x' has been turned on, so the\nvalues will be visible in the logs.\n\nhttps://public-inbox.org/git/20181018220106.GU19800@szeder.dev/\n\n\n> diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\n> index 7aef39a2fd..d2045b63a6 100755\n> --- a/ci/print-test-failures.sh\n> +++ b/ci/print-test-failures.sh\n> @@ -69,7 +69,7 @@ do\n> \tfi\n> done\n> \n> -if [ $combined_trash_size -gt 0 ]\n> +if [ -n \"$TRAVIS_JOB_ID\" -a $combined_trash_size -gt 0 ]\n\nNit: if [ -n \"$TRAVIS_JOB_ID\" ] && [ $combined_trash_size -gt 0 ]\n\nMore importantly: is this necessary, because on Azure Pipelines there\nis no URL from where the logs could be downloaded conveniently and\nreliably? I wonder whether it's worth to spend the extra effort in\npreparing the base64-encoded trash directories of failed tests in the\nfirst place.\n\n> then\n> \techo \"------------------------------------------------------------------------\"\n> \techo \"Trash directories embedded in this log can be extracted by running:\"\n"},{"id":"367711","messageId":"nycvar.QRO.7.76.6.1901261916340.45@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqzhrr54h0.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-26T18:37:40Z","receivedAt":"2019-01-26T18:37:56Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 23 Jan 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> > index fe65144152..bcdcc71592 100755\n> > --- a/ci/install-dependencies.sh\n> > +++ b/ci/install-dependencies.sh\n> > @@ -37,7 +37,8 @@ osx-clang|osx-gcc)\n> > \tbrew update --quiet\n> > \t# Uncomment this if you want to run perf tests:\n> > \t# brew install gnu-time\n> \n> Isn't this comment now stale?\n\nNo, not really. Actually, not at all.\n\n> It would be, under this new arrangement of the code, most natural for\n> Those who want to use gnu-time to arrange it to be somehow added to\n> $BREW_INSTALL_PACKAGES, no?\n\nThe purpose of BREW_INSTALL_PACKAGES is to list the packages necessary to\nbuild Git and run its test suite, and the only reason why this is no\nlonger a hard-coded list of packages is that it depends on the CI platform\n(or more concretely, on the available macOS agents of said CI platform)\nwhich packages need to be installed to do so.\n\nThe gnu-time package is not such a package, and it is unlikely to be\ndependent on the particular CI you want to use.\n\nSo this spot really is the appropriate place to say that you want to\ninstall gnu-time.\n\n> > -\tbrew install git-lfs gettext\n> > +\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n> > +\tbrew install $BREW_INSTALL_PACKAGES\n> \n> It is unclear how this relates to \"encapsulate Travis-specific\"; it\n> is guessable that perhaps only under Travis we'd use\n> BREW_INSTALL_PACKAGES, but it still is unexplained why under other\n> CIs we don't do lfs or gettext (if that is the plan---that is not\n> clear, either).\n\nI added this paragraph to the commit message:\n\n\tAlso, the configurations of the available agents are different. For\n\texample, Travis' and Azure Pipelines' macOS agents are set up\n\tdifferently, so that on Travis, we have to install the git-lfs and\n\tgettext Homebrew packages, and on Azure Pipelines we do not need to.\n\nDoes that sound good enough?\n\n> > \tbrew link --force gettext\n> > \tbrew install caskroom/cask/perforce\n> > \t;;\n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index c26bb6a274..4456dbbcb0 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -1,8 +1,26 @@\n> > # Library of functions shared by all CI scripts\n> > \n> > -# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n> > -# want here. We want the source branch instead.\n> > -CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> > +if test true = \"$TRAVIS\"\n> > +then\n> > +\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n> > +\t# what we want here. We want the source branch instead.\n> > +\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> > +\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n> > +\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n> > +\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n> > +\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n> > +\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n> > +\n> > +\tcache_dir=\"$HOME/travis-cache\"\n> > +\n> > +\turl_for_job_id () {\n> > +\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n> > +\t}\n> > +\n> > +\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n> > +\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n> > +\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n> > +fi\n> \n> OK. I actually was expecting this to be something like\n> \n> \tcase \"$CI_TYPE\" in\n> \tTravis) ... the reindented travis specific stuff here ... ;;\n> \t*) echo >&2 \"unknown CI_TYPE: $CI_TYPE\"; exit 1 ;;\n> \tesac\n> \n> to make it equally easy to add not just the second one but the third\n> one.\n\nThat's a very valid point.\n\nThere is not actually a CI_TYPE available, so I still use an `if`\nconstruct, but I added an `else` that will error out if we could not\nidentify the CI platform.\n\n> > @@ -28,7 +46,7 @@ skip_branch_tip_with_tag () {\n> > # job if we encounter the same tree again and can provide a useful info\n> > # message.\n> > save_good_tree () {\n> > -\techo \"$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID\" >>\"$good_trees_file\"\n> > +\techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n> \n> Makes sense. $CI_COMMIT is still coming from $TRAVIS_COMMIT but is\n> that different from \"git rev-parse $CI_BRANCH\" I wonder (thinking\n> aloud, not suggesting any changes).\n\nActually, I found out the hard way (a succeeding build that was a false\nnegative) that `TRAVIS_COMMIT` does, at least sometimes, refer to the\n*merge* commit merging e.g. `pu` into `master` (which does not make sense\nat all to me, because that's neither what you did nor what you asked\nTravis to do for you).\n\nSo no, we need to use `TRAVIS_COMMIT` here lest we record a tree that was\nnot tested.\n\n> > @@ -38,7 +56,7 @@ save_good_tree () {\n> > ...\n> > diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\n> > index 7aef39a2fd..d2045b63a6 100755\n> > --- a/ci/print-test-failures.sh\n> > +++ b/ci/print-test-failures.sh\n> > @@ -69,7 +69,7 @@ do\n> > \tfi\n> > done\n> > \n> > -if [ $combined_trash_size -gt 0 ]\n> > +if [ -n \"$TRAVIS_JOB_ID\" -a $combined_trash_size -gt 0 ]\n> \n> I do not quite understand this change.\n> \n> If this wants to say that \"we know how to show failures only when we\n> know the job id\", then testing for $CI_JOB_ID would be more\n> appropriate, and if this wants to say that \"we know how to show\n> failures only when we are running under travis\", then it would be\n> more appropriate to switch on test \"$TRAVIS\" = true that is used at\n> the beginning of ci/lib.sh (or if you would switch to \"case $CI_TYPE\"\n> there, use the same construct). I cannot quite make an argument to\n> support the use of $TRAVIS_JOB_ID in either case.\n\nI tried to be cheap and *not* introduce that CI_TYPE variable. But you are\ncorrect, it is the right thing to do.\n\n> > +test -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n> > gem install asciidoctor\n> > \n> > make check-builtins\n> \n> OK, BREW_INSTALL_PACKAGES has an assignment in the travis specific\n> section to trigger install of two separate packages, and there is no\n> assignment to ALREADY_HAVE_ASCIIDOCTOR in the travis specific\n> codepath, so we will end up installing asciidoctor here. Makes\n> sense, even though this one too is under-explained just like the\n> other one is.\n\nHopefully the added paragraph (which says \"for example\" and then uses the\nHomebrew packages as example) is enough to make the reader not puzzled\nabout this here change, too?\n\nCiao,\nDscho\n"},{"id":"367712","messageId":"nycvar.QRO.7.76.6.1901261938210.45@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqtvhz54gz.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 04/21] ci: inherit --jobs via MAKEFLAGS in run-build-and-tests","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-26T18:48:45Z","receivedAt":"2019-01-26T18:48:55Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 23 Jan 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > From: Johannes Schindelin \n> >\n> > Let's not decide in the generic ci/ script how many jobs to run in\n> > parallel; it is easy enough to hand that information down via the\n> > `MAKEFLAGS`.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > ci/run-build-and-tests.sh | 2 +-\n> > 1 file changed, 1 insertion(+), 1 deletion(-)\n> >\n> > diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\n> > index db342bb6a8..80d72d120f 100755\n> > --- a/ci/run-build-and-tests.sh\n> > +++ b/ci/run-build-and-tests.sh\n> > @@ -7,7 +7,7 @@\n> > \n> > ln -s \"$cache_dir/.prove\" t/.prove\n> > \n> > -make --jobs=2\n> > +make\n> > make --quiet test\n> > if test \"$jobname\" = \"linux-gcc\"\n> > then\n> \n> As there is no assignment to MAKEFLAGS in this patch, is it intended\n> for this step to change behaviour (possibly with the intention to\n> add \"default 2 jobs at least under travis\" back later in the\n> series)? Not that it matters too much, but it is unnerving to see\n> that the proposed log message promising \"it is easy enough\" while\n> not actually doing so, without expressing an intention.\n\nI was under the incorrect impression that Travis already configured a\nMAKEFLAGS=--jobs= by default (I got fooled by the GIT_PROVE_OPTS\nsetting that configures that --jobs option).\n\nBut the spirit of the change is still correct, I would think, so I made\nthe change more complete by actually setting MAKEFLAGS in the\nCI-specific sections, and by removing the explicit --jobs=2 parameters in\nthe scripts.\n\nCiao,\nDscho\n"},{"id":"367753","messageId":"nycvar.QRO.7.76.6.1901271547590.45@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqo98754gy.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 06/21] test-date: add a subcommand to measure times in shell scripts","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-27T14:54:20Z","receivedAt":"2019-01-27T14:54:28Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 23 Jan 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > From: Johannes Schindelin \n> >\n> > In the next commit, we want to teach Git's test suite to optionally\n> > output test results in JUnit-style .xml files. These files contain\n> > information about the time spent. So we need a way to measure time.\n> >\n> > While we could use `date +%s` for that, this will give us only seconds,\n> > i.e. very coarse-grained timings.\n> >\n> > GNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output),\n> > but there is no equivalent in BSD `date` (read: on macOS, we would not\n> > be able to obtain precise timings).\n> >\n> > So let's introduce `test-tool date getnanos`, with an optional start\n> > time, that outputs preciser values.\n> \n> I think the goal to have our own stopwatch so that we do not have to\n> worry about differences among system-provided ones makes sense.\n\nAnd so that we do not have to worry about all kinds of unportable shell\ncommands. That was my main motivation here, TBH.\n\n> The only thing that may become an issue is how widely available\n> getnanotime() is. As \"test-date\" itself is built on any platform an\n> end-user/developer runs our tests, which is wider set of platforms\n> than what we run Travis and other CIs on, unconditionally relying on\n> its availability might pose an issue. I dunno.\n\nWell, getnanotime() itself looks (I checked) at highres_nanos()' output,\nwhich can be 0 on unsuported platforms, and falls back to\ngettimeofday_nanos(). The worst that can happen there, as far as I can\ntell, is that the platform has a certain granularity (which is true of\nhighres_nanos(), too), and that granularity can be 1 second. Which is not\nvery good, but if that's the best the platform can provide, I am prepared\nto take it. Which is exactly what this patch does.\n\nDo you have any splendid idea how to improve this approach?\n\nCiao,\nDscho\n"},{"id":"367754","messageId":"nycvar.QRO.7.76.6.1901271555230.45@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqpnsn7zgr.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 10/21] ci: move the Windows job to the top","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-27T18:22:37Z","receivedAt":"2019-01-27T18:22:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 23 Jan 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > From: Johannes Schindelin \n> >\n> > The Windows job currently takes a whopping ~1h20m to complete. Which is\n> > *far* longer than the next-longest job takes (linux-gcc, ~35m). As such,\n> > it makes sense to start the Windows job first, to minimize the overall\n> > run time (which is now pretty safely the run time of the Windows job).\n> \n> Is the reason why Windows job gets started first is to make sure\n> that it, which is known to take the longest time, never has to wait\n> before starting while other jobs run, in case there is limited\n> parallelism?\n\nYes, in order to optimize the overall run time. Like, if you have N jobs,\nand you know that one of them takes longer than the other combined, it\nreally only makes sense to start that one as first one.\n\n> The last part of this sentence is what readers of this\n> step will need in order to be convinced by the justification given,\n> because (1) if the jobs run totally serially, the order does not\n> matter much---if anything, running shorter jobs first would give\n> results from more jobs sooner, and (2) if the jobs run totally in\n> parallel, the order does not matter as long as we have enough\n> parallelism.\n\nRight, I think I totally forgot to mention that Azure Pipelines offers 10\nparallel jobs to open source projects for free. Which means that we will\nrun up to 10 jobs in parallel, as long as no other build is running, of\ncourse.\n\nWill adjust the commit message.\n\n> > This commit is best viewed with `--color-moved`.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > azure-pipelines.yml | 172 ++++++++++++++++++++++----------------------\n> > 1 file changed, 86 insertions(+), 86 deletions(-)\n> \n> For those who are seeing this azure-pipelines series for the first\n> time, it would probably be unclear what the point of adding an\n> entire file in 09/21 and them moving lines around in 10/21 is. If\n> somebody asked me why, I wouldn't be able to explain why it is a\n> good idea.\n> \n> Would it hurt readability if these two steps are combined?\n\nThe thing is, I tried (of course) to replicate the Travis configuration as\nclosely as possible. And the Windows job was in a specific location there.\n\nHowever, I just realized that I *added* the Windows job to the Pipelines\nright from the start, and that is definitely not on par with Travis, as\nthe Travis configuration did not define a Windows job (instead it has code\nto trigger a dedicated Azure Pipeline).\n\nSo what I will do instead is to\n\n- *not* add the Windows-specific part in the commit that adds the initial\n Azure Pipelines support, and\n\n- explain in the commit message of that commit that the idea is to imitate\n our existing Travis configuration as closely as possible.\n\nCiao,\nDscho\n"},{"id":"367755","messageId":"nycvar.QRO.7.76.6.1901271925450.45@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqimyf54gx.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 08/21] ci/lib.sh: add support for Azure Pipelines","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-27T18:26:00Z","receivedAt":"2019-01-27T18:26:07Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 23 Jan 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > From: Johannes Schindelin \n> >\n> > This patch introduces a conditional arm that defines some environment\n> > variables and a function that displays the URL given the job id (to\n> > identify previous runs for known-good trees).\n> >\n> > For example, we do not have to install the git-lfs and gettext packages\n> > on Azure Pipelines' macOS agents: they are already installed, and trying\n> > to install them again would result in an error.\n> \n> The rationale in the second paragraph is shared with the earlier one\n> that introduced BREW_INSTALL_PACKAGES variable in the first place.\n> It would have made more sense if it was explained there (i.e. say\n> \"in environments other than Travis, some packages may be pre\n> installed and asking to install them again may cause failure, so\n> make it configurable what to install\")---that way, the explanation\n> in this step's log message can just say \"Because Azure pipeline's\n> macOS agents already have git-lfs and gettext installed, unlike\n> Travis, we can leave BREW_INSTALL_PACKAGES empty\".\n\nMakes sense.\n\nWill do that,\nDscho\n\n> \n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > ci/lib.sh | 23 +++++++++++++++++++++++\n> > 1 file changed, 23 insertions(+)\n> >\n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index 4456dbbcb0..a9eb4f4eae 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -20,6 +20,29 @@ then\n> > \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n> > \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n> > \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n> > +elif test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n> > +then\n> > +\t# We are running in Azure Pipelines\n> > +\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n> > +\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n> > +\tCI_JOB_ID=\"$BUILD_BUILDID\"\n> > +\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n> > +\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n> > +\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n> > +\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n> > +\tCC=\"${CC:-gcc}\"\n> > +\n> > +\t# use a subdirectory of the cache dir (because the file share is shared\n> > +\t# among *all* phases)\n> > +\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n> > +\n> > +\turl_for_job_id () {\n> > +\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n> > +\t}\n> > +\n> > +\tBREW_INSTALL_PACKAGES=\n> > +\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> > +\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n> > fi\n> > \n> > skip_branch_tip_with_tag () {\n> \n"},{"id":"367756","messageId":"nycvar.QRO.7.76.6.1901271927130.45@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqd0on54gw.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 09/21] Add a build definition for Azure DevOps","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-27T18:30:18Z","receivedAt":"2019-01-27T18:30:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 23 Jan 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > From: Johannes Schindelin \n> >\n> > This commit adds an azure-pipelines.yml file which is Azure DevOps'\n> > equivalent to Travis CI's .travis.yml.\n> >\n> > To make things a bit easier to understand, we refrain from using the\n> > `matrix` feature here because (while it is powerful) it can be a bit\n> > confusing to users who are not familiar with CI setups. Therefore, we\n> > use a separate phase even for similar configurations (such as GCC vs\n> > Clang on Linux, GCC vs Clang on macOS).\n> >\n> > Also, we make use of the shiny new feature we just introduced where the\n> > test suite can output JUnit-style .xml files. This information is made\n> > available in a nice UI that allows the viewer to filter by phase and/or\n> > test number, and to see trends such as: number of (failing) tests, time\n> > spent running the test suite, etc.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > azure-pipelines.yml | 306 ++++++++++++++++++++++++++++++++++++++++++\n> \n> Way-overlong line in this file bothers me somewhat, but let's say\n> these are not for human consumption and it is OK ;-)\n\nRight ;-)\n\n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index a9eb4f4eae..91cf1402bf 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -42,7 +42,7 @@ then\n> > \n> > \tBREW_INSTALL_PACKAGES=\n> > \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n> > -\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n> > +\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n> > fi\n> \n> Interesting to see this change here, not in the previous step. If\n> we stopped at 08/21, did we even have a chance to enter this elif\n> block? If not, it probably is sane to squash this hunk to the\n> previous one that introduced the elif block.\n\nOh my. You're right. I think I just picked the wrong commit to make this\nfixup.\n\nWill squash as suggested,\nDscho\n"},{"id":"367758","messageId":"nycvar.QRO.7.76.6.1901271938440.45@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqq1s5354gu.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 17/21] tests: add t/helper/ to the PATH with --with-dashes","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-27T18:40:34Z","receivedAt":"2019-01-27T18:40:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 23 Jan 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > From: Johannes Schindelin \n> >\n> > We really need to be able to find the test helpers... Really. This\n> > change was forgotten when we moved the test helpers into t/helper/\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > t/test-lib.sh | 2 +-\n> > 1 file changed, 1 insertion(+), 1 deletion(-)\n> \n> It is unfortunate that --with-dashes is needed for valgrind and\n> GIT_TEST_INSTALLED. Otherwise I do not see why we care about\n> keeping the option working.\n> \n> Does any or your test matrix entry actually use --with-dashes and\n> rely on it working, or is it just the stuff that implicitly depend\n> on it?\n\nThis here test matrix does not rely on it. But I do hope to get the\nBusyBox support (where we ship a subset of Git for Windows that is\nintended to be used by 3rd-party applications only, called \"MinGit\", with\nBusyBox instead of the MSYS2 Bash and all of those shell utilities\nrequired by Git's scripts) closer to production. And to test *that*, I\n*need* GIT_TEST_INSTALLED to work properly.\n\nCiao,\nDscho\n\n> Thanks.\n> \n> >\n> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> > index 9c0ca5effb..c790e98fd2 100644\n> > --- a/t/test-lib.sh\n> > +++ b/t/test-lib.sh\n> > @@ -1227,7 +1227,7 @@ else # normal case, use ../bin-wrappers only unless $with_dashes:\n> > \tGIT_EXEC_PATH=$GIT_BUILD_DIR\n> > \tif test -n \"$with_dashes\"\n> > \tthen\n> > -\t\tPATH=\"$GIT_BUILD_DIR:$PATH\"\n> > +\t\tPATH=\"$GIT_BUILD_DIR:$GIT_BUILD_DIR/t/helper:$PATH\"\n> > \tfi\n> > fi\n> > GIT_TEMPLATE_DIR=\"$GIT_BUILD_DIR\"/templates/blt\n> \n"},{"id":"367759","messageId":"nycvar.QRO.7.76.6.1901272003560.45@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqq7eev54gv.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 10/21] ci: move the Windows job to the top","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-27T19:05:53Z","receivedAt":"2019-01-27T19:06:06Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Wed, 23 Jan 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > From: Johannes Schindelin \n> >\n> > The Windows job currently takes a whopping ~1h20m to complete. Which is\n> > *far* longer than the next-longest job takes (linux-gcc, ~35m). As such,\n> > it makes sense to start the Windows job first, to minimize the overall\n> > run time (which is now pretty safely the run time of the Windows job).\n> \n> Is the reason why Windows job gets started first is to make sure\n> that it, which is known to take the longest time, never has to wait\n> before starting while other jobs run, in case there is limited\n> parallelism? The last part of this sentence is what readers of this\n> step will need in order to be convinced by the justification given,\n> because (1) if the jobs run totally serially, the order does not\n> matter much---if anything, running shorter jobs first would give\n> results from more jobs sooner, and (2) if the jobs run totally in\n> parallel, the order does not matter as long as we have enough\n> parallelism.\n\nSee my response to the other mail you sent (which seemed to be a first\ndraft of this mail I am replying to?).\n\n> > This commit is best viewed with `--color-moved`.\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > azure-pipelines.yml | 172 ++++++++++++++++++++++----------------------\n> > 1 file changed, 86 insertions(+), 86 deletions(-)\n> \n> For those who are seeing this azure-pipelines series for the first\n> time, it would probably be unclear what the point of adding an\n> entire file in 09/21 and them moving lines around in 10/21 is. If\n> somebody asked me why, I wouldn't be able to explain why it is a\n> good idea.\n> \n> The same comment applies to 11/21.\n> \n> Would it hurt readability if these steps are combined?\n> \n> If 09/21 were \"copy travis.yml to create a moral-equivalent set-up\n> for azure.yml\", then it is an entirely different story (i.e. \"we\n> start from an equivalent setup as we have, and then tweak to match\n> our needs better, and we can view the tweak easier as a separate\n> step\"), but I did not get the impression that it was what happened\n> there in 09/21.\n\nIndeed, that *was* the intention. I tried to clarify that, by just *not*\nadding the Windows-specific part in the commit that adds\nazure-pipelines.yml, as it really should imitate .travis.yml closely.\n\nCiao,\nDscho\n"},{"id":"367778","messageId":"nycvar.QRO.7.76.6.1901272022410.45@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"20190125135127.GE6702@szeder.dev","subject":"Re: [PATCH v4 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-27T21:22:23Z","receivedAt":"2019-01-27T21:22:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Gábor,\n\nOn Fri, 25 Jan 2019, SZEDER Gábor wrote:\n\n> On Wed, Jan 23, 2019 at 06:40:16AM -0800, Johannes Schindelin via\n> GitGitGadget wrote:\n> > From: Johannes Schindelin \n> > \n> > The upcoming patches will allow building git.git via Azure Pipelines\n> > (i.e. Azure DevOps' Continuous Integration), where variable names and\n> > URLs look a bit different than in Travis CI.\n> > \n> > Signed-off-by: Johannes Schindelin \n> > ---\n> \n> > diff --git a/ci/lib.sh b/ci/lib.sh\n> > index c26bb6a274..4456dbbcb0 100755\n> > --- a/ci/lib.sh\n> > +++ b/ci/lib.sh\n> > @@ -1,8 +1,26 @@\n> > # Library of functions shared by all CI scripts\n> > \n> > -# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n> > -# want here. We want the source branch instead.\n> > -CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> > +if test true = \"$TRAVIS\"\n> > +then\n> > +\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n> > +\t# what we want here. We want the source branch instead.\n> > +\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n> > +\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n> > +\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n> > +\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n> > +\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n> > +\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n> > +\n> > +\tcache_dir=\"$HOME/travis-cache\"\n> > +\n> > +\turl_for_job_id () {\n> > +\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n> > +\t}\n> > +\n> > +\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n> > +\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n> > +\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n> > +fi\n> \n> Please set these variables after 'set -x' has been turned on, so the\n> values will be visible in the logs.\n> \n> https://public-inbox.org/git/20181018220106.GU19800@szeder.dev/\n\nOooops. Sorry, I overlooked this! I'm sorry. Will fix.\n\nThe fix will actually be squashed into 1/21 (\"travis: fix skipping tagged\nreleases\") because due to the re-ordering requested by Junio, this is now\nthe first offender.\n\n> > diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\n> > index 7aef39a2fd..d2045b63a6 100755\n> > --- a/ci/print-test-failures.sh\n> > +++ b/ci/print-test-failures.sh\n> > @@ -69,7 +69,7 @@ do\n> > \tfi\n> > done\n> > \n> > -if [ $combined_trash_size -gt 0 ]\n> > +if [ -n \"$TRAVIS_JOB_ID\" -a $combined_trash_size -gt 0 ]\n> \n> Nit: if [ -n \"$TRAVIS_JOB_ID\" ] && [ $combined_trash_size -gt 0 ]\n> \n> More importantly: is this necessary, because on Azure Pipelines there\n> is no URL from where the logs could be downloaded conveniently and\n> reliably? I wonder whether it's worth to spend the extra effort in\n> preparing the base64-encoded trash directories of failed tests in the\n> first place.\n\nYou're right, of course. There *should* be a way to extract these from the\nlog even on Azure Pipelines. I did find the API to get the logs, but it\nseems that at least for the moment, there is no environment variable\nspecifying the number of the current task (on Azure Pipelines, the logs\nare split by task). So I could not actually figure out any way to generate\nthe Azure Pipelines equivalent of this URL:\n\n\thttps://api.travis-ci.org/v3/job/$TRAVIS_JOB_ID/log.txt\n\nBut then, this code (which builds tar files of the trash directories of\nthe failed tests and then logs a base64 version of it) simply is a\nworkaround in the first place, accommodating for the fact that Travis does\nnot offer build artifacts. But Azure Pipelines does. So that's what I am\ndoing with those trash directories now: publish them as build artifacts\n(which can be downloaded conveniently as .zip files from the web page\nshowing the build result).\n\nThanks for helping me improve this patch series!\nDscho\n\n> > then\n> > \techo \"------------------------------------------------------------------------\"\n> > \techo \"Trash directories embedded in this log can be extracted by running:\"\n> \n> "},{"id":"367781","messageId":"xmqqsgxdelqy.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"xmqqo98754gy.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 06/21] test-date: add a subcommand to measure times in shell scripts","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-27T23:14:45Z","receivedAt":"2019-01-27T23:14:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> I think the goal to have our own stopwatch so that we do not have to\n> worry about differences among system-provided ones makes sense.\n>\n> The only thing that may become an issue is how widely available\n> getnanotime() is. As \"test-date\" itself is built on any platform an\n> end-user/developer runs our tests, which is wider set of platforms\n> than what we run Travis and other CIs on, unconditionally relying on\n> its availability might pose an issue.\n\nSorry for a false alarm, as the codebase in many places like\nfsmonitor, progress, trace and wt-status have been assuming\ngetnanotime() to be available for quite some time, and this is just\nanother user of the same function.\n\n"},{"id":"367782","messageId":"xmqqlg35elgv.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1901261916340.45@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v4 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-27T23:20:48Z","receivedAt":"2019-01-27T23:20:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> No, not really. Actually, not at all.\n>\n>> It would be, under this new arrangement of the code, most natural for\n>> Those who want to use gnu-time to arrange it to be somehow added to\n>> $BREW_INSTALL_PACKAGES, no?\n>\n> The purpose of BREW_INSTALL_PACKAGES is to list the packages necessary to\n> build Git and run its test suite, and the only reason why this is no\n> longer a hard-coded list of packages is that it depends on the CI platform\n> (or more concretely, on the available macOS agents of said CI platform)\n> which packages need to be installed to do so.\n>\n> The gnu-time package is not such a package, and it is unlikely to be\n> dependent on the particular CI you want to use.\n\nThose who want to do perf tests in the current setup would need to\ninstall gnu-time because the current setup is only Travis, whose\nmacOS agent does not have it preinstalled. Other CI platforms'\nmacOS agents may already have it, they may not want to get an error\nby trying to install it there. I am not sure how that is different\nfrom the situation for gettext etc.?\n\n\n\n\n\n"},{"id":"367783","messageId":"f553fd4fb37c26e04b404b0819d3f7bcd51cd4f2.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 01/21] travis: fix skipping tagged releases","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:49Z","receivedAt":"2019-01-27T23:26:54Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWhen building a PR, TRAVIS_BRANCH refers to the *target branch*.\nTherefore, if a PR targets `master`, and `master` happened to be tagged,\nwe skipped the build by mistake.\n\nFix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*)\nwhen available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also\nknown as \"push builds\").\n\nLet's give it a new variable name, too: CI_BRANCH (as it is different\nfrom TRAVIS_BRANCH). This also prepares for the upcoming patches which\nwill make our ci/* code a bit more independent from Travis and open it\nto other CI systems (in particular to Azure Pipelines).\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib-travisci.sh | 25 ++++++++++++++-----------\n 1 file changed, 14 insertions(+), 11 deletions(-)\n\ndiff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh\nindex 69dff4d1ec..9c6ddeb374 100755\n--- a/ci/lib-travisci.sh\n+++ b/ci/lib-travisci.sh\n@@ -5,18 +5,17 @@ skip_branch_tip_with_tag () {\n \t# at the same commit as the tip of the branch is pushed, and building\n \t# both at the same time is a waste.\n \t#\n-\t# Travis gives a tagname e.g. v2.14.0 in $TRAVIS_BRANCH when\n-\t# the build is triggered by a push to a tag. Let's see if\n-\t# $TRAVIS_BRANCH is exactly at a tag, and if so, if it is\n-\t# different from $TRAVIS_BRANCH. That way, we can tell if\n-\t# we are building the tip of a branch that is tagged and\n-\t# we can skip the build because we won't be skipping a build\n-\t# of a tag.\n-\n-\tif TAG=$(git describe --exact-match \"$TRAVIS_BRANCH\" 2>/dev/null) &&\n-\t\ttest \"$TAG\" != \"$TRAVIS_BRANCH\"\n+\t# When the build is triggered by a push to a tag, $CI_BRANCH will\n+\t# have that tagname, e.g. v2.14.0. Let's see if $CI_BRANCH is\n+\t# exactly at a tag, and if so, if it is different from $CI_BRANCH.\n+\t# That way, we can tell if we are building the tip of a branch that\n+\t# is tagged and we can skip the build because we won't be skipping a\n+\t# build of a tag.\n+\n+\tif TAG=$(git describe --exact-match \"$CI_BRANCH\" 2>/dev/null) &&\n+\t\ttest \"$TAG\" != \"$CI_BRANCH\"\n \tthen\n-\t\techo \"$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)\"\n+\t\techo \"$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)\"\n \t\texit 0\n \tfi\n }\n@@ -81,6 +80,10 @@ check_unignored_build_artifacts ()\n # and installing dependencies.\n set -ex\n \n+# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n+# want here. We want the source branch instead.\n+CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+\n cache_dir=\"$HOME/travis-cache\"\n good_trees_file=\"$cache_dir/good-trees\"\n \n-- \ngitgitgadget\n\n"},{"id":"367784","messageId":"f0852de8abe2a501fdb829b1c241f02fa1deeaf0.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 02/21] ci: rename the library of common functions","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:50Z","receivedAt":"2019-01-27T23:26:55Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe name is hard-coded to reflect that we use Travis CI for continuous\ntesting.\n\nIn the next commits, we will extend this to be able use Azure DevOps,\ntoo.\n\nSo let's adjust the name to make it more generic.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 2 +-\n ci/{lib-travisci.sh => lib.sh} | 0\n ci/print-test-failures.sh | 2 +-\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 2 +-\n 8 files changed, 7 insertions(+), 7 deletions(-)\n rename ci/{lib-travisci.sh => lib.sh} (100%)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 06c3546e1e..fe65144152 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -3,7 +3,7 @@\n # Install dependencies required to build and test Git on Linux and macOS\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\ndiff --git a/ci/lib-travisci.sh b/ci/lib.sh\nsimilarity index 100%\nrename from ci/lib-travisci.sh\nrename to ci/lib.sh\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex d55460a212..7aef39a2fd 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -3,7 +3,7 @@\n # Print output of failing tests\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n # Tracing executed commands would produce too much noise in the loop below.\n set +x\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex cda170d5c2..db342bb6a8 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -3,7 +3,7 @@\n # Build and test Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n ln -s \"$cache_dir/.prove\" t/.prove\n \ndiff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh\nindex 21637903ce..751acfcf8a 100755\n--- a/ci/run-linux32-docker.sh\n+++ b/ci/run-linux32-docker.sh\n@@ -3,7 +3,7 @@\n # Download and run Docker image to build and test 32-bit Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n docker pull daald/ubuntu32:xenial\n \ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\nindex 5688f261d0..dc189c7456 100755\n--- a/ci/run-static-analysis.sh\n+++ b/ci/run-static-analysis.sh\n@@ -3,7 +3,7 @@\n # Perform various static code analysis checks\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n make --jobs=2 coccicheck\n \ndiff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh\nindex d99a180e52..a73a4eca0a 100755\n--- a/ci/run-windows-build.sh\n+++ b/ci/run-windows-build.sh\n@@ -6,7 +6,7 @@\n # supported) and a commit hash.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n test $# -ne 2 && echo \"Unexpected number of parameters\" && exit 1\n test -z \"$GFW_CI_TOKEN\" && echo \"GFW_CI_TOKEN not defined\" && exit\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex a20de9ca12..d3cdbac73f 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -3,7 +3,7 @@\n # Perform sanity checks on documentation and build it.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n gem install asciidoctor\n \n-- \ngitgitgadget\n\n"},{"id":"367785","messageId":"pull.31.v5.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v4.git.gitgitgadget@gmail.com","subject":"[PATCH v5 00/21] Offer to run CI/PR builds in Azure Pipelines","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:48Z","receivedAt":"2019-01-27T23:26:56Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"For a long time already, we have tested Git's source code continuously via\nTravis CI, see e.g. https://travis-ci.org/git/git/builds/421738884. It has\nserved us well, and more and more developers actually pay attention and\nbenefit from the testing this gives us.\n\nIt is also an invaluable tool for contributors who can validate their code\ncontributions via PRs on GitHub, e.g. to verify that their tests do actually\nrun on macOS (i.e. with the BSD family of Unix tools instead of the GNU\none).\n\nThe one sad part about this is the Windows support. Travis used to lack it\n(Travis announced some early support for Windows, hot off the press\n[https://blog.travis-ci.com/2018-10-11-windows-early-release]), and we work\naround that by using Azure Pipelines (the CI part of Azure DevOps, formerly\nknown as Visual Studio Team Services) indirectly: one phase in Travis would\ntrigger a build, wait for its log, and then paste that log.\n\nAs Git's Windows builds (and tests!) take quite a bit of time, Travis often\ntimed out, or somehow the trigger did not work, and for security reasons\n(the Windows builds are performed in a private pool of containers), the\nWindows builds are completely disabled for Pull Requests on GitHub.\n\nOne might ask why we did not use Azure Pipelines directly. There were a\ncouple of reasons for that:\n\n * most notably, Azure Pipelines' build logs could not be viewed\n anonymously,\n * while Azure Pipelines had Linux and Windows agents, it lacked macOS\n agents,\n * etc\n\nThe main two reasons no longer apply: macOS agents are available now\n[https://docs.microsoft.com/en-us/azure/devops/release-notes/2018/jul-10-vsts]\n, and are public projects\n[https://docs.microsoft.com/en-us/azure/devops/organizations/public/about-public-projects] \nnow, i.e. it is possible to configure a Azure Pipelines project so that \nanybody can view the logs. Since I offered v1, Azure Pipelines has been made\navailable via the GitHub Marketplace, free of cost for open source projects.\n\nI had secured such a public project for Git for Windows already, and I also\ngot one for Git. For now, the latter is hooked up with my personal git.git\nfork on GitHub, but it is my hope that I convince y'all that these Azure\nPipelines builds are a good idea, and then hook it up with \nhttps://github.com/git/git.\n\nAs a special treat, this patch series adds the ability to present the\noutcome of Git's test suite as JUnit-style .xml files. This allows the Azure\nPipelines build to present fun diagrams, trends, and makes it a lot easier\nto drill down to test failures than before. See for example \nhttps://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details\n[https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details] \n(you can click on the label of the failed test, and then see the detailed\noutput in the right pane).\n\nBut maybe you're not interested as much in the presentation of test failures\nas in the speed? Well, I got you covered with that, too. As of v3, the test\nsuite is run in parallel on Windows, cutting the overall run time to ~33\nminutes (see one of the latest builds\n[https://dev.azure.com/git/git/_build/results?buildId=311&view=logs], for\nexample).\n\nThis patch series took waaaaaaaay more time than I had originally\nanticipated, but I think that in particular the advanced display of the test\nresults and the reduction of the overall run time was worth it. Please let\nme know what you think about this.\n\nChanges since v4:\n\n * Clarified in the commit message why BREW_INSTALL_PACKAGES is needed.\n * Added a clause to error out if we could not detect the CI type.\n * Introduced a CI_TYPE variable to guard CI type-specific code.\n * MAKEFLAGS is now actually configured in the CI type-specific section.\n * Moved the inadvertent fixup for GIT_TEST_OPTS from \"Add a build\n definition for Azure DevOps\" to \"ci/lib.sh: add support for Azure\n Pipelines\".\n * Backed out the Windows support from the commit adding the initial Azure\n Pipelines support, instead adding it later to help the flow of the patch\n series.\n * Clarified in \"test-date: add a subcommand to measure times in shell\n scripts\"'s commit message that one of the goals is portability.\n * Moved all of those CI type-specific definitions in ci/lib.sh after the \n set -ex statement (as suggested by Gábor a loooong time ago already).\n * Guarded the Travis-specific way to \"publish\" trash directories of failed\n tests via base64-encoded tar files in the log; For Azure Pipelines, these\n are published as build artifacts instead.\n\nChanges since v3:\n\n * Fixed the incorrect usage of cut -c that resulted in an empty trace when\n failed tests were published (except if the very first test case in a test\n script failed, then it was correct by mistake).\n * Excluded the previous test case's \"ok\" (or \"not ok\") line from the trace\n published with failed tests.\n * Renamed TRAVIS_BRANCH to CI_BRANCH already in the first commit, as we\n should not override TRAVIS_BRANCH with something that it is not.\n * Rebased onto current master to avoid merge conflicts with the\n recently-merged sg/stress-test branch (no, Junio, I really trust myself\n more than you to resolve those merge conflicts).\n\nChanges since v2:\n\n * Removed left-over debugging code that would skip a good chunk of \n t0000-init.sh.\n * Fixed the URL of the build badge.\n * Removed a trailing empty line from, and added a missing closing pointy\n bracket to, ci/mount-fileshare.sh.\n * Moved the \"travis: fix skipping tagged releases\" commit up to the\n beginning of the patch series.\n * The commit message of \"ci/lib.sh: add support for Azure Pipelines\" now\n mentions explicitly that the Homebrew packages that need to be installed\n on Travis' macOS agents are already installed on Azure Pipelines'.\n * Some commands were not guarded by || exit 1, i.e. if they would fail, the\n build would not have failed.\n * We now install gcc-8 for the linux-gcc job.\n * We no longer try to re-run failed tests with verbose log. Instead, we\n simply use the verbose log to begin with. Tests showed that it had a\n negligible impact on the overall run time.\n * The test_atexit_handler function was scratched; It would be the right\n thing to do, but is actually an independent topic (it was only\n implemented in v2 to accommodate the \"re-run with verbose log on failure\"\n trick)\n * We now use a new YAML schema (most notably, \"phase\" is now known as\n \"job\")\n * The Windows job contained PowerShell sections that were indented with 3\n spaces instead of 2.\n * The Windows job is now structured better, by separating different\n concerns into different \"tasks\" so that it is easier to see what exactly\n failed (was it the build? was it the test?)\n * The Windows job was split into a job to build Git and 10 parallel jobs to\n run the test suite with the artifacts built by the first job. This\n reduces the overall run time from ~1h20 (which was the run time by the\n Windows job) to ~35 minutes (which is the run time of the linux-gcc job).\n * The JUnit XML is now written using a test helper rather than a fragile\n and probably not even portable sed call.\n * Since we needed to determine the file size of the verbose log (to cut out\n individual test cases' log output), we now introduce a test helper to do\n that, and use it throughout the test suite (where Perl was used before).\n * It would appear that a recent change either in Cygwin or in the Azure VMs\n causes problems sporadically where the trash directories cannot be\n removed, but a subsequent rm will succeed. We now simply do that, because\n it won't harm the common case (where the first rm succeeds already) and\n because it helps the Windows job succeed pretty reliably.\n\nChanges since v1:\n\n * Removed a superfluous eval.\n * Added the commit that fixes the Travis PR builds targeting master that \n just happens to be tagged (see \n https://travis-ci.org/git/git/jobs/424276413 for an incorrectly-skipped\n build).\n * The commit messages and the cover letter now reflect the name change from\n Visual Studio Team Services to Azure DevOps (and in particular, Azure\n Pipelines for the automated builds).\n * Now we're using test_atexit (which we introduced for that purpose)\n instead of hard-coding kill_p4d and stop_git_daemon.\n * The build should now also succeed for Pull Requests (where secret\n variables are not available, for security reasons, and as a consequence\n the file share cannot be mounted).\n * The shell scripted parts now use proper && chains.\n\nJohannes Schindelin (21):\n travis: fix skipping tagged releases\n ci: rename the library of common functions\n ci/lib.sh: encapsulate Travis-specific things\n ci: inherit --jobs via MAKEFLAGS in run-build-and-tests\n ci: use a junction on Windows instead of a symlink\n test-date: add a subcommand to measure times in shell scripts\n tests: optionally write results as JUnit-style .xml\n ci/lib.sh: add support for Azure Pipelines\n Add a build definition for Azure DevOps\n ci: add a Windows job to the Azure Pipelines definition\n ci: use git-sdk-64-minimal build artifact\n mingw: be more generous when wrapping up the setitimer() emulation\n README: add a build badge (status of the Azure Pipelines build)\n tests: avoid calling Perl just to determine file sizes\n tests: include detailed trace logs with --write-junit-xml upon failure\n mingw: try to work around issues with the test cleanup\n tests: add t/helper/ to the PATH with --with-dashes\n t0061: fix with --with-dashes and RUNTIME_PREFIX\n tests: optionally skip bin-wrappers/\n ci: speed up Windows phase\n ci: parallelize testing on Windows\n\n Makefile | 11 +\n README.md | 2 +\n azure-pipelines.yml | 387 ++++++++++++++++++++++++++++\n ci/install-dependencies.sh | 5 +-\n ci/{lib-travisci.sh => lib.sh} | 92 +++++--\n ci/make-test-artifacts.sh | 12 +\n ci/mount-fileshare.sh | 25 ++\n ci/print-test-failures.sh | 15 +-\n ci/run-build-and-tests.sh | 9 +-\n ci/run-linux32-build.sh | 2 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 4 +-\n ci/run-test-slice.sh | 17 ++\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 7 +-\n compat/mingw.c | 2 +-\n t/.gitignore | 1 +\n t/README | 9 +\n t/helper/test-date.c | 12 +\n t/helper/test-path-utils.c | 64 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/helper/test-xml-encode.c | 80 ++++++\n t/t0021-conversion.sh | 2 +-\n t/t0061-run-command.sh | 3 +-\n t/t1050-large.sh | 2 +-\n t/t5315-pack-objects-compression.sh | 2 +-\n t/t9303-fast-import-compression.sh | 2 +-\n t/test-lib.sh | 138 +++++++++-\n 29 files changed, 862 insertions(+), 49 deletions(-)\n create mode 100644 azure-pipelines.yml\n rename ci/{lib-travisci.sh => lib.sh} (50%)\n create mode 100755 ci/make-test-artifacts.sh\n create mode 100755 ci/mount-fileshare.sh\n create mode 100755 ci/run-test-slice.sh\n create mode 100644 t/helper/test-xml-encode.c\n\n\nbase-commit: 16a465bc018d09e9d7bbbdc5f40a7fb99c21f8ef\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-31%2Fdscho%2Fvsts-ci-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-31/dscho/vsts-ci-v5\nPull-Request: https://github.com/gitgitgadget/git/pull/31\n\nRange-diff vs v4:\n\n 1: 6ed2f2a35c ! 1: f553fd4fb3 travis: fix skipping tagged releases\n @@ -21,14 +21,6 @@\n --- a/ci/lib-travisci.sh\n +++ b/ci/lib-travisci.sh\n @@\n - # Library of functions shared by all CI scripts\n - \n -+# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n -+# want here. We want the source branch instead.\n -+CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n -+\n - skip_branch_tip_with_tag () {\n - \t# Sometimes, a branch is pushed at the same time the tag that points\n \t# at the same commit as the tip of the branch is pushed, and building\n \t# both at the same time is a waste.\n \t#\n @@ -57,3 +49,14 @@\n \t\texit 0\n \tfi\n }\n +@@\n + # and installing dependencies.\n + set -ex\n + \n ++# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n ++# want here. We want the source branch instead.\n ++CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n ++\n + cache_dir=\"$HOME/travis-cache\"\n + good_trees_file=\"$cache_dir/good-trees\"\n + \n 2: b086f36660 = 2: f0852de8ab ci: rename the library of common functions\n 3: 2d0b62f186 ! 3: 7c16d31b46 ci/lib.sh: encapsulate Travis-specific things\n @@ -6,6 +6,18 @@\n (i.e. Azure DevOps' Continuous Integration), where variable names and\n URLs look a bit different than in Travis CI.\n \n + Also, the configurations of the available agents are different. For\n + example, Travis' and Azure Pipelines' macOS agents are set up\n + differently, so that on Travis, we have to install the git-lfs and\n + gettext Homebrew packages, and on Azure Pipelines we do not need to.\n + Likewise, Azure Pipelines' Ubuntu agents already have asciidoctor\n + installed.\n +\n + Finally, on Azure Pipelines the natural way is not to base64-encode tar\n + files of the trash directories of failed tests, but to publish build\n + artifacts instead. Therefore, that code to log those base64-encoded tar\n + files is guarded to be Travis-specific.\n +\n Signed-off-by: Johannes Schindelin \n \n diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n @@ -25,36 +37,6 @@\n diff --git a/ci/lib.sh b/ci/lib.sh\n --- a/ci/lib.sh\n +++ b/ci/lib.sh\n -@@\n - # Library of functions shared by all CI scripts\n - \n --# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n --# want here. We want the source branch instead.\n --CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n -+if test true = \"$TRAVIS\"\n -+then\n -+\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n -+\t# what we want here. We want the source branch instead.\n -+\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n -+\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n -+\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n -+\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n -+\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n -+\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n -+\n -+\tcache_dir=\"$HOME/travis-cache\"\n -+\n -+\turl_for_job_id () {\n -+\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n -+\t}\n -+\n -+\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n -+\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n -+\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n -+fi\n - \n - skip_branch_tip_with_tag () {\n - \t# Sometimes, a branch is pushed at the same time the tag that points\n @@\n # job if we encounter the same tree again and can provide a useful info\n # message.\n @@ -100,6 +82,35 @@\n # and installing dependencies.\n set -ex\n \n +-# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n +-# want here. We want the source branch instead.\n +-CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n ++if test true = \"$TRAVIS\"\n ++then\n ++\tCI_TYPE=travis\n ++\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n ++\t# what we want here. We want the source branch instead.\n ++\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n ++\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n ++\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n ++\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n ++\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n ++\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n ++\n ++\tcache_dir=\"$HOME/travis-cache\"\n ++\n ++\turl_for_job_id () {\n ++\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n ++\t}\n ++\n ++\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n ++\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n ++\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n ++else\n ++\techo \"Could not identify CI type\" >&2\n ++\texit 1\n ++fi\n + \n -cache_dir=\"$HOME/travis-cache\"\n good_trees_file=\"$cache_dir/good-trees\"\n \n @@ -124,14 +135,20 @@\n --- a/ci/print-test-failures.sh\n +++ b/ci/print-test-failures.sh\n @@\n - \tfi\n - done\n - \n --if [ $combined_trash_size -gt 0 ]\n -+if [ -n \"$TRAVIS_JOB_ID\" -a $combined_trash_size -gt 0 ]\n - then\n - \techo \"------------------------------------------------------------------------\"\n - \techo \"Trash directories embedded in this log can be extracted by running:\"\n + \t\ttest_name=\"${TEST_EXIT%.exit}\"\n + \t\ttest_name=\"${test_name##*/}\"\n + \t\ttrash_dir=\"trash directory.$test_name\"\n ++\t\tcase \"$CI_TYPE\" in\n ++\t\ttravis)\n ++\t\t\t;;\n ++\t\t*)\n ++\t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n ++\t\t\texit 1\n ++\t\t\t;;\n ++\t\tesac\n + \t\ttrash_tgz_b64=\"trash.$test_name.base64\"\n + \t\tif [ -d \"$trash_dir\" ]\n + \t\tthen\n \n diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\n --- a/ci/test-documentation.sh\n 4: 83b92a87e7 < -: ---------- ci: inherit --jobs via MAKEFLAGS in run-build-and-tests\n -: ---------- > 4: bf72fb1004 ci: inherit --jobs via MAKEFLAGS in run-build-and-tests\n 5: 6c9eb4e33e = 5: 681f8e65b4 ci: use a junction on Windows instead of a symlink\n 6: aa053ed993 ! 6: ccf8bf53d7 test-date: add a subcommand to measure times in shell scripts\n @@ -14,7 +14,10 @@\n be able to obtain precise timings).\n \n So let's introduce `test-tool date getnanos`, with an optional start\n - time, that outputs preciser values.\n + time, that outputs preciser values. Note that this might not actually\n + give us nanosecond precision on some platforms, but it will give us as\n + precise information as possible, without the portability issues of shell\n + commands.\n \n Granted, it is a bit pointless to try measuring times accurately in\n shell scripts, certainly to nanosecond precision. But it is better than\n 7: 07fb37e0a6 = 7: 31fbe11425 tests: optionally write results as JUnit-style .xml\n 8: 178dc9b789 ! 8: 47fe8aa84a ci/lib.sh: add support for Azure Pipelines\n @@ -6,9 +6,12 @@\n variables and a function that displays the URL given the job id (to\n identify previous runs for known-good trees).\n \n - For example, we do not have to install the git-lfs and gettext packages\n - on Azure Pipelines' macOS agents: they are already installed, and trying\n - to install them again would result in an error.\n + Because Azure Pipeline's macOS agents already have git-lfs and gettext\n + installed, we can leave `BREW_INSTALL_PACKAGES` empty (unlike in\n + Travis' case).\n +\n + Note: this patch does not introduce an Azure Pipelines definition yet;\n + That is left for the next patch.\n \n Signed-off-by: Johannes Schindelin \n \n @@ -16,11 +19,12 @@\n --- a/ci/lib.sh\n +++ b/ci/lib.sh\n @@\n - \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n + \texport MAKEFLAGS=\"--jobs=2\"\n +elif test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n +then\n ++\tCI_TYPE=azure-pipelines\n +\t# We are running in Azure Pipelines\n +\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n +\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n @@ -41,7 +45,24 @@\n +\n +\tBREW_INSTALL_PACKAGES=\n +\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n -+\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n - fi\n - \n - skip_branch_tip_with_tag () {\n ++\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n ++\texport MAKEFLAGS=\"--jobs=10\"\n + else\n + \techo \"Could not identify CI type\" >&2\n + \texit 1\n +\n + diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\n + --- a/ci/print-test-failures.sh\n + +++ b/ci/print-test-failures.sh\n +@@\n + \t\tcase \"$CI_TYPE\" in\n + \t\ttravis)\n + \t\t\t;;\n ++\t\tazure-pipelines)\n ++\t\t\tmkdir -p failed-test-artifacts\n ++\t\t\tmv \"$trash_dir\" failed-test-artifacts\n ++\t\t\tcontinue\n ++\t\t\t;;\n + \t\t*)\n + \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n + \t\t\texit 1\n 9: eaba471b89 ! 9: 6044b919b0 Add a build definition for Azure DevOps\n @@ -5,6 +5,19 @@\n This commit adds an azure-pipelines.yml file which is Azure DevOps'\n equivalent to Travis CI's .travis.yml.\n \n + The main idea is to replicate the Travis configuration as faithfully as\n + possible, to make it easy to compare the Azure Pipeline builds to the\n + Travis ones (spoiler: some parts, especially the macOS jobs, are way\n + faster in Azure Pileines). Meaning: the number and the order of the jobs\n + added in this commit faithfully replicates what we have in .travis.yml.\n +\n + Note: Our .travis.yml configuration has a Windows part that is *not*\n + replicated in the Azure Pipelines definition. The reason is easy to see:\n + As Travis cannot support our Windws needs (even with the preliminary\n + Windows support that was recently added to Travis after waiting for\n + *years* for that feature, our test suite would simply hit Travis'\n + timeout every single time).\n +\n To make things a bit easier to understand, we refrain from using the\n `matrix` feature here because (while it is powerful) it can be a bit\n confusing to users who are not familiar with CI setups. Therefore, we\n @@ -15,7 +28,9 @@\n test suite can output JUnit-style .xml files. This information is made\n available in a nice UI that allows the viewer to filter by phase and/or\n test number, and to see trends such as: number of (failing) tests, time\n - spent running the test suite, etc.\n + spent running the test suite, etc. (While this seemingly contradicts the\n + intention to replicate the Travis configuration as faithfully as\n + possible, it is just too nice to show off that capability here already.)\n \n Signed-off-by: Johannes Schindelin \n \n @@ -60,6 +75,12 @@\n + platform: Linux\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n ++ - task: PublishBuildArtifacts@1\n ++ displayName: 'Publish trash directories of failed tests'\n ++ condition: failed()\n ++ inputs:\n ++ PathtoPublish: t/failed-test-artifacts\n ++ ArtifactName: failed-test-artifacts\n +\n +- job: linux_gcc\n + displayName: linux-gcc\n @@ -91,6 +112,12 @@\n + platform: Linux\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n ++ - task: PublishBuildArtifacts@1\n ++ displayName: 'Publish trash directories of failed tests'\n ++ condition: failed()\n ++ inputs:\n ++ PathtoPublish: t/failed-test-artifacts\n ++ ArtifactName: failed-test-artifacts\n +\n +- job: osx_clang\n + displayName: osx-clang\n @@ -120,6 +147,12 @@\n + platform: macOS\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n ++ - task: PublishBuildArtifacts@1\n ++ displayName: 'Publish trash directories of failed tests'\n ++ condition: failed()\n ++ inputs:\n ++ PathtoPublish: t/failed-test-artifacts\n ++ ArtifactName: failed-test-artifacts\n +\n +- job: osx_gcc\n + displayName: osx-gcc\n @@ -147,6 +180,12 @@\n + platform: macOS\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n ++ - task: PublishBuildArtifacts@1\n ++ displayName: 'Publish trash directories of failed tests'\n ++ condition: failed()\n ++ inputs:\n ++ PathtoPublish: t/failed-test-artifacts\n ++ ArtifactName: failed-test-artifacts\n +\n +- job: gettext_poison\n + displayName: GETTEXT_POISON\n @@ -178,92 +217,12 @@\n + platform: Linux\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n -+\n -+- job: windows\n -+ displayName: Windows\n -+ condition: succeeded()\n -+ pool: Hosted\n -+ timeoutInMinutes: 240\n -+ steps:\n -+ - powershell: |\n -+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n -+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n -+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n -+ }\n -+ displayName: 'Mount test-cache'\n -+ env:\n -+ GITFILESHAREPWD: $(gitfileshare.pwd)\n -+ - powershell: |\n -+ # Helper to check the error level of the latest command (exit with error when appropriate)\n -+ function c() { if (!$?) { exit(1) } }\n -+\n -+ # Add build agent's MinGit to PATH\n -+ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n -+\n -+ # Helper to initialize (or update) a Git worktree\n -+ function init ($path, $url, $set_origin) {\n -+ if (Test-Path $path) {\n -+ cd $path; c\n -+ if (Test-Path .git) {\n -+ & git init; c\n -+ } else {\n -+ & git status\n -+ }\n -+ } else {\n -+ & git init $path; c\n -+ cd $path; c\n -+ }\n -+ & git config core.autocrlf false; c\n -+ & git config core.untrackedCache true; c\n -+ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n -+ & git remote add origin $url; c\n -+ }\n -+ & git fetch --depth=1 $url master; c\n -+ & git reset --hard FETCH_HEAD; c\n -+ & git clean -df; c\n -+ }\n -+\n -+ # Initialize Git for Windows' SDK\n -+ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n -+ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n -+\n -+ # Let Git ignore the SDK and the test-cache\n -+ \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n -+ displayName: 'Initialize the Git for Windows SDK'\n -+ - powershell: |\n -+ & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n -+ export MAKEFLAGS=-j10\n -+ export DEVELOPER=1\n -+ export NO_PERL=1\n -+ export NO_SVN_TESTS=1\n -+ export GIT_TEST_SKIP_REBASE_P=1\n -+\n -+ ci/run-build-and-tests.sh || {\n -+ ci/print-test-failures.sh\n -+ exit 1\n -+ }\n -+ \"@\n -+ if (!$?) { exit(1) }\n -+ displayName: 'Build & Test'\n -+ env:\n -+ HOME: $(Build.SourcesDirectory)\n -+ MSYSTEM: MINGW64\n -+ - powershell: |\n -+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n -+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n -+ }\n -+ displayName: 'Unmount test-cache'\n -+ condition: true\n -+ env:\n -+ GITFILESHAREPWD: $(gitfileshare.pwd)\n -+ - task: PublishTestResults@2\n -+ displayName: 'Publish Test Results **/TEST-*.xml'\n ++ - task: PublishBuildArtifacts@1\n ++ displayName: 'Publish trash directories of failed tests'\n ++ condition: failed()\n + inputs:\n -+ mergeTestResults: true\n -+ testRunTitle: 'windows'\n -+ platform: Windows\n -+ publishRunAttachments: false\n -+ condition: succeededOrFailed()\n ++ PathtoPublish: t/failed-test-artifacts\n ++ ArtifactName: failed-test-artifacts\n +\n +- job: linux32\n + displayName: Linux32\n @@ -273,11 +232,14 @@\n + - bash: |\n + test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n +\n -+ sudo AGENT_OS=\"$AGENT_OS\" BUILD_BUILDNUMBER=\"$BUILD_BUILDNUMBER\" BUILD_REPOSITORY_URI=\"$BUILD_REPOSITORY_URI\" BUILD_SOURCEBRANCH=\"$BUILD_SOURCEBRANCH\" BUILD_SOURCEVERSION=\"$BUILD_SOURCEVERSION\" SYSTEM_PHASENAME=\"$SYSTEM_PHASENAME\" SYSTEM_TASKDEFINITIONSURI=\"$SYSTEM_TASKDEFINITIONSURI\" SYSTEM_TEAMPROJECT=\"$SYSTEM_TEAMPROJECT\" CC=$CC MAKEFLAGS=-j3 bash -lxc ci/run-linux32-docker.sh || exit 1\n ++ res=0\n ++ sudo AGENT_OS=\"$AGENT_OS\" BUILD_BUILDNUMBER=\"$BUILD_BUILDNUMBER\" BUILD_REPOSITORY_URI=\"$BUILD_REPOSITORY_URI\" BUILD_SOURCEBRANCH=\"$BUILD_SOURCEBRANCH\" BUILD_SOURCEVERSION=\"$BUILD_SOURCEVERSION\" SYSTEM_PHASENAME=\"$SYSTEM_PHASENAME\" SYSTEM_TASKDEFINITIONSURI=\"$SYSTEM_TASKDEFINITIONSURI\" SYSTEM_TEAMPROJECT=\"$SYSTEM_TEAMPROJECT\" CC=$CC MAKEFLAGS=\"$MAKEFLAGS\" bash -lxc ci/run-linux32-docker.sh || res=1\n +\n + sudo chmod a+r t/out/TEST-*.xml\n ++ test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts\n +\n -+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n ++ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || res=1\n ++ exit $res\n + displayName: 'ci/run-linux32-docker.sh'\n + env:\n + GITFILESHAREPWD: $(gitfileshare.pwd)\n @@ -289,6 +251,12 @@\n + platform: Linux\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n ++ - task: PublishBuildArtifacts@1\n ++ displayName: 'Publish trash directories of failed tests'\n ++ condition: failed()\n ++ inputs:\n ++ PathtoPublish: t/failed-test-artifacts\n ++ ArtifactName: failed-test-artifacts\n +\n +- job: static_analysis\n + displayName: StaticAnalysis\n @@ -331,19 +299,6 @@\n + env:\n + GITFILESHAREPWD: $(gitfileshare.pwd)\n \n - diff --git a/ci/lib.sh b/ci/lib.sh\n - --- a/ci/lib.sh\n - +++ b/ci/lib.sh\n -@@\n - \n - \tBREW_INSTALL_PACKAGES=\n - \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n --\texport GIT_TEST_OPTS=\"--quiet --write-junit-xml\"\n -+\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n - fi\n - \n - skip_branch_tip_with_tag () {\n -\n diff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\n new file mode 100755\n --- /dev/null\n 10: 5bdc6a08a8 ! 10: 3a2993c826 ci: move the Windows job to the top\n @@ -1,17 +1,30 @@\n Author: Johannes Schindelin \n \n - ci: move the Windows job to the top\n + ci: add a Windows job to the Azure Pipelines definition\n \n - The Windows job currently takes a whopping ~1h20m to complete. Which is\n - *far* longer than the next-longest job takes (linux-gcc, ~35m). As such,\n - it makes sense to start the Windows job first, to minimize the overall\n - run time (which is now pretty safely the run time of the Windows job).\n + Previously, we did not have robust support for Windows in our CI\n + definition, simply because Travis cannot accommodate our needs (even\n + after Travis added experimental Windows support very recently, it takes\n + longer than Travis' 50 minute timeout to build Git and run the test\n + suite on Windows). Instead, we used a hack that started a dedicated\n + Azure Pipeline from Travis and waited for the output, often timing out\n + (which is quite fragile, as we found out).\n \n - This affects only the Azure Pipelines configuration, not the Travis one,\n - of course, as Travis cannot run our Windows job: 1h20m is distinctly\n - longer than the 50 minute timeout of Travis' free tier.\n + With this commit, we finally have first-class support for Windows in our\n + CI definition (in the Azure Pipelines one, that is).\n \n - This commit is best viewed with `--color-moved`.\n + Due to our reliance on Unix shell scripting in the test suite, combined\n + with the challenges on executing such scripts on Windows, the Windows\n + job currently takes a whopping ~1h20m to complete. Which is *far* longer\n + than the next-longest job takes (linux-gcc, ~35m).\n +\n + Now, Azure Pipelines's free tier for open source projects (such as Git)\n + offers up to 10 concurrent jobs for free, meaning that the overall run\n + time will be dominated by the slowest job(s).\n +\n + Therefore, it makes sense to start the Windows job first, to minimize\n + the time the entire build takes from start to end (which is now pretty\n + safely the run time of the Windows job).\n \n Signed-off-by: Johannes Schindelin \n \n @@ -75,7 +88,6 @@\n + displayName: 'Initialize the Git for Windows SDK'\n + - powershell: |\n + & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n -+ export MAKEFLAGS=-j10\n + export DEVELOPER=1\n + export NO_PERL=1\n + export NO_SVN_TESTS=1\n @@ -107,100 +119,13 @@\n + platform: Windows\n + publishRunAttachments: false\n + condition: succeededOrFailed()\n ++ - task: PublishBuildArtifacts@1\n ++ displayName: 'Publish trash directories of failed tests'\n ++ condition: failed()\n ++ inputs:\n ++ PathtoPublish: t/failed-test-artifacts\n ++ ArtifactName: failed-test-artifacts\n +\n - job: linux_clang\n displayName: linux-clang\n condition: succeeded()\n -@@\n - publishRunAttachments: false\n - condition: succeededOrFailed()\n - \n --- job: windows\n -- displayName: Windows\n -- condition: succeeded()\n -- pool: Hosted\n -- timeoutInMinutes: 240\n -- steps:\n -- - powershell: |\n -- if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n -- net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n -- cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n -- }\n -- displayName: 'Mount test-cache'\n -- env:\n -- GITFILESHAREPWD: $(gitfileshare.pwd)\n -- - powershell: |\n -- # Helper to check the error level of the latest command (exit with error when appropriate)\n -- function c() { if (!$?) { exit(1) } }\n --\n -- # Add build agent's MinGit to PATH\n -- $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n --\n -- # Helper to initialize (or update) a Git worktree\n -- function init ($path, $url, $set_origin) {\n -- if (Test-Path $path) {\n -- cd $path; c\n -- if (Test-Path .git) {\n -- & git init; c\n -- } else {\n -- & git status\n -- }\n -- } else {\n -- & git init $path; c\n -- cd $path; c\n -- }\n -- & git config core.autocrlf false; c\n -- & git config core.untrackedCache true; c\n -- if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n -- & git remote add origin $url; c\n -- }\n -- & git fetch --depth=1 $url master; c\n -- & git reset --hard FETCH_HEAD; c\n -- & git clean -df; c\n -- }\n --\n -- # Initialize Git for Windows' SDK\n -- $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n -- init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n --\n -- # Let Git ignore the SDK and the test-cache\n -- \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n -- displayName: 'Initialize the Git for Windows SDK'\n -- - powershell: |\n -- & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n -- export MAKEFLAGS=-j10\n -- export DEVELOPER=1\n -- export NO_PERL=1\n -- export NO_SVN_TESTS=1\n -- export GIT_TEST_SKIP_REBASE_P=1\n --\n -- ci/run-build-and-tests.sh || {\n -- ci/print-test-failures.sh\n -- exit 1\n -- }\n -- \"@\n -- if (!$?) { exit(1) }\n -- displayName: 'Build & Test'\n -- env:\n -- HOME: $(Build.SourcesDirectory)\n -- MSYSTEM: MINGW64\n -- - powershell: |\n -- if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n -- cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n -- }\n -- displayName: 'Unmount test-cache'\n -- condition: true\n -- env:\n -- GITFILESHAREPWD: $(gitfileshare.pwd)\n -- - task: PublishTestResults@2\n -- displayName: 'Publish Test Results **/TEST-*.xml'\n -- inputs:\n -- mergeTestResults: true\n -- testRunTitle: 'windows'\n -- platform: Windows\n -- publishRunAttachments: false\n -- condition: succeededOrFailed()\n --\n - - job: linux32\n - displayName: Linux32\n - condition: succeeded()\n 11: 93eebb74ce ! 11: 52d63018c9 ci: use git-sdk-64-minimal build artifact\n @@ -78,6 +78,6 @@\n - powershell: |\n - & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n + & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n - export MAKEFLAGS=-j10\n export DEVELOPER=1\n export NO_PERL=1\n + export NO_SVN_TESTS=1\n 12: 18d76823e5 = 12: 1cef14e4c0 mingw: be more generous when wrapping up the setitimer() emulation\n 13: 5d593acee5 = 13: eda46863e8 README: add a build badge (status of the Azure Pipelines build)\n 14: 3083041e33 = 14: 4303bbd4e8 tests: avoid calling Perl just to determine file sizes\n 15: 1c44d3b8b8 = 15: dfd3516f39 tests: include detailed trace logs with --write-junit-xml upon failure\n 16: 59c1194ae2 = 16: 9984c6d17b mingw: try to work around issues with the test cleanup\n 17: 966c412f03 = 17: a8101ae3e1 tests: add t/helper/ to the PATH with --with-dashes\n 18: d613c79aff = 18: 7518266a19 t0061: fix with --with-dashes and RUNTIME_PREFIX\n 19: a198885b76 = 19: 6a8adbe8ba tests: optionally skip bin-wrappers/\n 20: 6520f4603c < -: ---------- ci: speed up Windows phase\n -: ---------- > 20: b39e16527d ci: speed up Windows phase\n 21: 8bdd9804a1 ! 21: 6ff87d1772 ci: parallelize testing on Windows\n @@ -78,7 +78,6 @@ $^\n displayName: 'Download git-sdk-64-minimal'\n - powershell: |\n & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n -- export MAKEFLAGS=-j10\n - export DEVELOPER=1\n - export NO_PERL=1\n - export NO_SVN_TESTS=1\n @@ -90,7 +89,6 @@ $^\n + env:\n + HOME: $(Build.SourcesDirectory)\n + MSYSTEM: MINGW64\n -+ MAKEFLAGS: -j10\n + DEVELOPER: 1\n + NO_PERL: 1\n + - task: PublishPipelineArtifact@0\n @@ -162,7 +160,6 @@ $^\n env:\n HOME: $(Build.SourcesDirectory)\n MSYSTEM: MINGW64\n -+ MAKEFLAGS: -j10\n + NO_SVN_TESTS: 1\n + GIT_TEST_SKIP_REBASE_P: 1\n - powershell: |\n\n-- \ngitgitgadget\n"},{"id":"367788","messageId":"7c16d31b46258ad00768c6125d0c6ad64d341c15.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:51Z","receivedAt":"2019-01-27T23:26:59Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe upcoming patches will allow building git.git via Azure Pipelines\n(i.e. Azure DevOps' Continuous Integration), where variable names and\nURLs look a bit different than in Travis CI.\n\nAlso, the configurations of the available agents are different. For\nexample, Travis' and Azure Pipelines' macOS agents are set up\ndifferently, so that on Travis, we have to install the git-lfs and\ngettext Homebrew packages, and on Azure Pipelines we do not need to.\nLikewise, Azure Pipelines' Ubuntu agents already have asciidoctor\ninstalled.\n\nFinally, on Azure Pipelines the natural way is not to base64-encode tar\nfiles of the trash directories of failed tests, but to publish build\nartifacts instead. Therefore, that code to log those base64-encoded tar\nfiles is guarded to be Travis-specific.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 3 ++-\n ci/lib.sh | 45 +++++++++++++++++++++++++++-----------\n ci/print-test-failures.sh | 8 +++++++\n ci/test-documentation.sh | 1 +\n 4 files changed, 43 insertions(+), 14 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex fe65144152..bcdcc71592 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -37,7 +37,8 @@ osx-clang|osx-gcc)\n \tbrew update --quiet\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\n-\tbrew install git-lfs gettext\n+\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n+\tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \tbrew install caskroom/cask/perforce\n \t;;\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 9c6ddeb374..3f286d86a6 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -24,7 +24,7 @@ skip_branch_tip_with_tag () {\n # job if we encounter the same tree again and can provide a useful info\n # message.\n save_good_tree () {\n-\techo \"$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID\" >>\"$good_trees_file\"\n+\techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n \t# limit the file size\n \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n \tmv \"$good_trees_file\".tmp \"$good_trees_file\"\n@@ -34,7 +34,7 @@ save_good_tree () {\n # successfully before (e.g. because the branch got rebased, changing only\n # the commit messages).\n skip_good_tree () {\n-\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $TRAVIS_COMMIT^{tree}) \" \"$good_trees_file\")\"\n+\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n \tthen\n \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n \t\t# Continue the build job.\n@@ -44,18 +44,18 @@ skip_good_tree () {\n \techo \"$good_tree_info\" | {\n \t\tread tree prev_good_commit prev_good_job_number prev_good_job_id\n \n-\t\tif test \"$TRAVIS_JOB_ID\" = \"$prev_good_job_id\"\n+\t\tif test \"$CI_JOB_ID\" = \"$prev_good_job_id\"\n \t\tthen\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit has already been built and tested successfully by this build job.\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\telse\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.\n-\t\t\tThe log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id\n+\t\t\tThe log of that build job is available at $(url_for_job_id $prev_good_job_id)\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\tfi\n@@ -80,11 +80,32 @@ check_unignored_build_artifacts ()\n # and installing dependencies.\n set -ex\n \n-# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n-# want here. We want the source branch instead.\n-CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+if test true = \"$TRAVIS\"\n+then\n+\tCI_TYPE=travis\n+\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n+\t# what we want here. We want the source branch instead.\n+\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n+\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n+\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n+\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n+\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n+\n+\tcache_dir=\"$HOME/travis-cache\"\n+\n+\turl_for_job_id () {\n+\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+else\n+\techo \"Could not identify CI type\" >&2\n+\texit 1\n+fi\n \n-cache_dir=\"$HOME/travis-cache\"\n good_trees_file=\"$cache_dir/good-trees\"\n \n mkdir -p \"$cache_dir\"\n@@ -94,13 +115,11 @@ skip_good_tree\n \n if test -z \"$jobname\"\n then\n-\tjobname=\"$TRAVIS_OS_NAME-$CC\"\n+\tjobname=\"$CI_OS_NAME-$CC\"\n fi\n \n export DEVELOPER=1\n export DEFAULT_TEST_TARGET=prove\n-export GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n-export GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n export GIT_TEST_CLONE_2GB=YesPlease\n if [ \"$jobname\" = linux-gcc ]; then\n \texport CC=gcc-8\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 7aef39a2fd..cf321b474d 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -38,6 +38,14 @@ do\n \t\ttest_name=\"${TEST_EXIT%.exit}\"\n \t\ttest_name=\"${test_name##*/}\"\n \t\ttrash_dir=\"trash directory.$test_name\"\n+\t\tcase \"$CI_TYPE\" in\n+\t\ttravis)\n+\t\t\t;;\n+\t\t*)\n+\t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n+\t\t\texit 1\n+\t\t\t;;\n+\t\tesac\n \t\ttrash_tgz_b64=\"trash.$test_name.base64\"\n \t\tif [ -d \"$trash_dir\" ]\n \t\tthen\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex d3cdbac73f..7d0beb2832 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -5,6 +5,7 @@\n \n . ${0%/*}/lib.sh\n \n+test -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n gem install asciidoctor\n \n make check-builtins\n-- \ngitgitgadget\n\n"},{"id":"367786","messageId":"31fbe114256688710216ad268c0c34206f71badd.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 07/21] tests: optionally write results as JUnit-style .xml","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:54Z","receivedAt":"2019-01-27T23:27:00Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis will come in handy when publishing the results of Git's test suite\nduring an automated Azure DevOps run.\n\nNote: we need to make extra sure that invalid UTF-8 encoding is turned\ninto valid UTF-8 (using the Replacement Character, \\uFFFD) because\nt9902's trace contains such invalid byte sequences, and the task in the\nAzure Pipeline that uploads the test results would refuse to do anything\nif it was asked to parse an .xml file with invalid UTF-8 in it.\n\nSigned-off-by: Johannes Schindelin \n---\n Makefile | 1 +\n t/.gitignore | 1 +\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/helper/test-xml-encode.c | 80 +++++++++++++++++++++++++++++++++\n t/test-lib.sh | 91 ++++++++++++++++++++++++++++++++++++++\n 6 files changed, 175 insertions(+)\n create mode 100644 t/helper/test-xml-encode.c\n\ndiff --git a/Makefile b/Makefile\nindex 1a44c811aa..044b4f77bd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -754,6 +754,7 @@ TEST_BUILTINS_OBJS += test-submodule-config.o\n TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o\n TEST_BUILTINS_OBJS += test-subprocess.o\n TEST_BUILTINS_OBJS += test-urlmatch-normalization.o\n+TEST_BUILTINS_OBJS += test-xml-encode.o\n TEST_BUILTINS_OBJS += test-wildmatch.o\n TEST_BUILTINS_OBJS += test-windows-named-pipe.o\n TEST_BUILTINS_OBJS += test-write-cache.o\ndiff --git a/t/.gitignore b/t/.gitignore\nindex 348715f0e4..91cf5772fe 100644\n--- a/t/.gitignore\n+++ b/t/.gitignore\n@@ -2,3 +2,4 @@\n /test-results\n /.prove\n /chainlinttmp\n+/out/\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex bfb195b1a8..4b4b397d93 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -49,6 +49,7 @@ static struct test_cmd cmds[] = {\n \t{ \"submodule-nested-repo-config\", cmd__submodule_nested_repo_config },\n \t{ \"subprocess\", cmd__subprocess },\n \t{ \"urlmatch-normalization\", cmd__urlmatch_normalization },\n+\t{ \"xml-encode\", cmd__xml_encode },\n \t{ \"wildmatch\", cmd__wildmatch },\n #ifdef GIT_WINDOWS_NATIVE\n \t{ \"windows-named-pipe\", cmd__windows_named_pipe },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 042f12464b..c0ab65e370 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -45,6 +45,7 @@ int cmd__submodule_config(int argc, const char **argv);\n int cmd__submodule_nested_repo_config(int argc, const char **argv);\n int cmd__subprocess(int argc, const char **argv);\n int cmd__urlmatch_normalization(int argc, const char **argv);\n+int cmd__xml_encode(int argc, const char **argv);\n int cmd__wildmatch(int argc, const char **argv);\n #ifdef GIT_WINDOWS_NATIVE\n int cmd__windows_named_pipe(int argc, const char **argv);\ndiff --git a/t/helper/test-xml-encode.c b/t/helper/test-xml-encode.c\nnew file mode 100644\nindex 0000000000..367c4875e6\n--- /dev/null\n+++ b/t/helper/test-xml-encode.c\n@@ -0,0 +1,80 @@\n+#include \"test-tool.h\"\n+\n+static const char *utf8_replace_character = \"�\";\n+\n+/*\n+ * Encodes (possibly incorrect) UTF-8 on to , to be embedded\n+ * in an XML file.\n+ */\n+int cmd__xml_encode(int argc, const char **argv)\n+{\n+\tunsigned char buf[1024], tmp[4], *tmp2 = NULL;\n+\tssize_t cur = 0, len = 1, remaining = 0;\n+\tunsigned char ch;\n+\n+\tfor (;;) {\n+\t\tif (++cur == len) {\n+\t\t\tlen = xread(0, buf, sizeof(buf));\n+\t\t\tif (!len)\n+\t\t\t\treturn 0;\n+\t\t\tif (len < 0)\n+\t\t\t\tdie_errno(\"Could not read \");\n+\t\t\tcur = 0;\n+\t\t}\n+\t\tch = buf[cur];\n+\n+\t\tif (tmp2) {\n+\t\t\tif ((ch & 0xc0) != 0x80) {\n+\t\t\t\tfputs(utf8_replace_character, stdout);\n+\t\t\t\ttmp2 = 0;\n+\t\t\t\tcur--;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\t*tmp2 = ch;\n+\t\t\ttmp2++;\n+\t\t\tif (--remaining == 0) {\n+\t\t\t\tfwrite(tmp, tmp2 - tmp, 1, stdout);\n+\t\t\t\ttmp2 = 0;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!(ch & 0x80)) {\n+\t\t\t/* 0xxxxxxx */\n+\t\t\tif (ch == '&')\n+\t\t\t\tfputs(\"&\", stdout);\n+\t\t\telse if (ch == '\\'')\n+\t\t\t\tfputs(\"'\", stdout);\n+\t\t\telse if (ch == '\"')\n+\t\t\t\tfputs(\""\", stdout);\n+\t\t\telse if (ch == '<')\n+\t\t\t\tfputs(\"<\", stdout);\n+\t\t\telse if (ch == '>')\n+\t\t\t\tfputs(\">\", stdout);\n+\t\t\telse if (ch >= 0x20)\n+\t\t\t\tfputc(ch, stdout);\n+\t\t\telse if (ch == 0x09 || ch == 0x0a || ch == 0x0d)\n+\t\t\t\tfprintf(stdout, \"&#x%02x;\", ch);\n+\t\t\telse\n+\t\t\t\tfputs(utf8_replace_character, stdout);\n+\t\t} else if ((ch & 0xe0) == 0xc0) {\n+\t\t\t/* 110XXXXx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 1;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else if ((ch & 0xf0) == 0xe0) {\n+\t\t\t/* 1110XXXX 10Xxxxxx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 2;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else if ((ch & 0xf8) == 0xf0) {\n+\t\t\t/* 11110XXX 10XXxxxx 10xxxxxx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 3;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else\n+\t\t\tfputs(utf8_replace_character, stdout);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a1abb1177a..a3b2166cb5 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -139,6 +139,9 @@ do\n \t\tverbose_log=t\n \t\ttee=t\n \t\t;;\n+\t--write-junit-xml)\n+\t\twrite_junit_xml=t\n+\t\t;;\n \t--stress)\n \t\tstress=t ;;\n \t--stress=*)\n@@ -622,11 +625,24 @@ trap 'exit $?' INT TERM HUP\n # the test_expect_* functions instead.\n \n test_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$*\"\n+\tfi\n \ttest_success=$(($test_success + 1))\n \tsay_color \"\" \"ok $test_count - $@\"\n }\n \n test_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_insert=\"\"\n+\t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n+\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\tjunit_insert=\"$junit_insert\"\n+\t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n+\tfi\n \ttest_failure=$(($test_failure + 1))\n \tsay_color error \"not ok $test_count - $1\"\n \tshift\n@@ -635,11 +651,19 @@ test_failure_ () {\n }\n \n test_known_broken_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (breakage fixed)\"\n+\tfi\n \ttest_fixed=$(($test_fixed+1))\n \tsay_color error \"ok $test_count - $@ # TODO known breakage vanished\"\n }\n \n test_known_broken_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (known breakage)\"\n+\tfi\n \ttest_broken=$(($test_broken+1))\n \tsay_color warn \"not ok $test_count - $@ # TODO known breakage\"\n }\n@@ -897,6 +921,10 @@ test_start_ () {\n \ttest_count=$(($test_count+1))\n \tmaybe_setup_verbose\n \tmaybe_setup_valgrind\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_start=$(test-tool date getnanos)\n+\tfi\n }\n \n test_finish_ () {\n@@ -934,6 +962,13 @@ test_skip () {\n \n \tcase \"$to_skip\" in\n \tt)\n+\t\tif test -n \"$write_junit_xml\"\n+\t\tthen\n+\t\t\tmessage=\"$(xml_attr_encode \"$skipped_reason\")\"\n+\t\t\twrite_junit_xml_testcase \"$1\" \\\n+\t\t\t\t\" \"\n+\t\tfi\n+\n \t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n@@ -949,9 +984,51 @@ test_at_end_hook_ () {\n \t:\n }\n \n+write_junit_xml () {\n+\tcase \"$1\" in\n+\t--truncate)\n+\t\t>\"$junit_xml_path\"\n+\t\tjunit_have_testcase=\n+\t\tshift\n+\t\t;;\n+\tesac\n+\tprintf '%s\\n' \"$@\" >>\"$junit_xml_path\"\n+}\n+\n+xml_attr_encode () {\n+\tprintf '%s\\n' \"$@\" | test-tool xml-encode\n+}\n+\n+write_junit_xml_testcase () {\n+\tjunit_attrs=\"name=\\\"$(xml_attr_encode \"$this_test.$test_count $1\")\\\"\"\n+\tshift\n+\tjunit_attrs=\"$junit_attrs classname=\\\"$this_test\\\"\"\n+\tjunit_attrs=\"$junit_attrs time=\\\"$(test-tool \\\n+\t\tdate getnanos $junit_start)\\\"\"\n+\twrite_junit_xml \"$(printf '%s\\n' \\\n+\t\t\" \" \"$@\" \" \")\"\n+\tjunit_have_testcase=t\n+}\n+\n test_done () {\n \tGIT_EXIT_OK=t\n \n+\tif test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n+\tthen\n+\t\ttest -n \"$junit_have_testcase\" || {\n+\t\t\tjunit_start=$(test-tool date getnanos)\n+\t\t\twrite_junit_xml_testcase \"all tests skipped\"\n+\t\t}\n+\n+\t\t# adjust the overall time\n+\t\tjunit_time=$(test-tool date getnanos $junit_suite_start)\n+\t\tsed \"s/]*/& time=\\\"$junit_time\\\"/\" \\\n+\t\t\t<\"$junit_xml_path\" >\"$junit_xml_path.new\"\n+\t\tmv \"$junit_xml_path.new\" \"$junit_xml_path\"\n+\n+\t\twrite_junit_xml \" \" \"\"\n+\tfi\n+\n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n \t\tmkdir -p \"$TEST_RESULTS_DIR\"\n@@ -1178,6 +1255,7 @@ then\n else\n \tmkdir -p \"$TRASH_DIRECTORY\"\n fi\n+\n # Use -P to resolve symlinks in our working directory so that the cwd\n # in subprocesses like git equals our $PWD (for pathname comparisons).\n cd -P \"$TRASH_DIRECTORY\" || exit 1\n@@ -1191,6 +1269,19 @@ then\n \ttest_done\n fi\n \n+if test -n \"$write_junit_xml\"\n+then\n+\tjunit_xml_dir=\"$TEST_OUTPUT_DIRECTORY/out\"\n+\tmkdir -p \"$junit_xml_dir\"\n+\tjunit_xml_base=${0##*/}\n+\tjunit_xml_path=\"$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml\"\n+\tjunit_attrs=\"name=\\\"${junit_xml_base%.sh}\\\"\"\n+\tjunit_attrs=\"$junit_attrs timestamp=\\\"$(TZ=UTC \\\n+\t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n+\twrite_junit_xml --truncate \"\" \" \"\n+\tjunit_suite_start=$(test-tool date getnanos)\n+fi\n+\n # Provide an implementation of the 'yes' utility\n yes () {\n \tif test $# = 0\n-- \ngitgitgadget\n\n"},{"id":"367787","messageId":"bf72fb10049283cb0d85ab3906f1ef535cb64ff8.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 04/21] ci: inherit --jobs via MAKEFLAGS in run-build-and-tests","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:52Z","receivedAt":"2019-01-27T23:27:00Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nLet's not decide in the generic ci/ part how many jobs to run in\nparallel; different CI configurations would favor a different number of\nparallel jobs, and it is easy enough to hand that information down via\nthe `MAKEFLAGS` variable.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 1 +\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-build.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/test-documentation.sh | 4 ++--\n 5 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 3f286d86a6..32a28fd209 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -101,6 +101,7 @@ then\n \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+\texport MAKEFLAGS=\"--jobs=2\"\n else\n \techo \"Could not identify CI type\" >&2\n \texit 1\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex db342bb6a8..80d72d120f 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -7,7 +7,7 @@\n \n ln -s \"$cache_dir/.prove\" t/.prove\n \n-make --jobs=2\n+make\n make --quiet test\n if test \"$jobname\" = \"linux-gcc\"\n then\ndiff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\nindex 2c60d2e70a..09e9276e12 100755\n--- a/ci/run-linux32-build.sh\n+++ b/ci/run-linux32-build.sh\n@@ -55,6 +55,6 @@ linux32 --32bit i386 su -m -l $CI_USER -c '\n \tset -ex\n \tcd /usr/src/git\n \ttest -n \"$cache_dir\" && ln -s \"$cache_dir/.prove\" t/.prove\n-\tmake --jobs=2\n+\tmake\n \tmake --quiet test\n '\ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\nindex dc189c7456..a19aa7ebbc 100755\n--- a/ci/run-static-analysis.sh\n+++ b/ci/run-static-analysis.sh\n@@ -5,7 +5,7 @@\n \n . ${0%/*}/lib.sh\n \n-make --jobs=2 coccicheck\n+make coccicheck\n \n set +x\n \ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex 7d0beb2832..be3b7d376a 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -12,7 +12,7 @@ make check-builtins\n make check-docs\n \n # Build docs with AsciiDoc\n-make --jobs=2 doc > >(tee stdout.log) 2> >(tee stderr.log >&2)\n+make doc > >(tee stdout.log) 2> >(tee stderr.log >&2)\n ! test -s stderr.log\n test -s Documentation/git.html\n test -s Documentation/git.xml\n@@ -24,7 +24,7 @@ check_unignored_build_artifacts\n \n # Build docs with AsciiDoctor\n make clean\n-make --jobs=2 USE_ASCIIDOCTOR=1 doc > >(tee stdout.log) 2> >(tee stderr.log >&2)\n+make USE_ASCIIDOCTOR=1 doc > >(tee stdout.log) 2> >(tee stderr.log >&2)\n sed '/^GIT_VERSION = / d' stderr.log\n ! test -s stderr.log\n test -s Documentation/git.html\n-- \ngitgitgadget\n\n"},{"id":"367789","messageId":"47fe8aa84aa11b8da9c01c5e8e88b68909dcfdbd.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 08/21] ci/lib.sh: add support for Azure Pipelines","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:55Z","receivedAt":"2019-01-27T23:27:01Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis patch introduces a conditional arm that defines some environment\nvariables and a function that displays the URL given the job id (to\nidentify previous runs for known-good trees).\n\nBecause Azure Pipeline's macOS agents already have git-lfs and gettext\ninstalled, we can leave `BREW_INSTALL_PACKAGES` empty (unlike in\nTravis' case).\n\nNote: this patch does not introduce an Azure Pipelines definition yet;\nThat is left for the next patch.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 25 +++++++++++++++++++++++++\n ci/print-test-failures.sh | 5 +++++\n 2 files changed, 30 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 32a28fd209..5505776876 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -102,6 +102,31 @@ then\n \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n \texport MAKEFLAGS=\"--jobs=2\"\n+elif test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n+then\n+\tCI_TYPE=azure-pipelines\n+\t# We are running in Azure Pipelines\n+\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n+\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n+\tCI_JOB_ID=\"$BUILD_BUILDID\"\n+\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n+\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n+\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n+\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n+\tCC=\"${CC:-gcc}\"\n+\n+\t# use a subdirectory of the cache dir (because the file share is shared\n+\t# among *all* phases)\n+\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n+\n+\turl_for_job_id () {\n+\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n+\texport MAKEFLAGS=\"--jobs=10\"\n else\n \techo \"Could not identify CI type\" >&2\n \texit 1\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex cf321b474d..e688a26f0d 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -41,6 +41,11 @@ do\n \t\tcase \"$CI_TYPE\" in\n \t\ttravis)\n \t\t\t;;\n+\t\tazure-pipelines)\n+\t\t\tmkdir -p failed-test-artifacts\n+\t\t\tmv \"$trash_dir\" failed-test-artifacts\n+\t\t\tcontinue\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n \t\t\texit 1\n-- \ngitgitgadget\n\n"},{"id":"367790","messageId":"ccf8bf53d7f168e3edea5e4a0cc5e15486abba22.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 06/21] test-date: add a subcommand to measure times in shell scripts","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:54Z","receivedAt":"2019-01-27T23:27:03Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIn the next commit, we want to teach Git's test suite to optionally\noutput test results in JUnit-style .xml files. These files contain\ninformation about the time spent. So we need a way to measure time.\n\nWhile we could use `date +%s` for that, this will give us only seconds,\ni.e. very coarse-grained timings.\n\nGNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output),\nbut there is no equivalent in BSD `date` (read: on macOS, we would not\nbe able to obtain precise timings).\n\nSo let's introduce `test-tool date getnanos`, with an optional start\ntime, that outputs preciser values. Note that this might not actually\ngive us nanosecond precision on some platforms, but it will give us as\nprecise information as possible, without the portability issues of shell\ncommands.\n\nGranted, it is a bit pointless to try measuring times accurately in\nshell scripts, certainly to nanosecond precision. But it is better than\nsecond-granularity.\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-date.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex a0837371ab..792a805374 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -7,6 +7,7 @@ static const char *usage_msg = \"\\n\"\n \" test-tool date parse [date]...\\n\"\n \" test-tool date approxidate [date]...\\n\"\n \" test-tool date timestamp [date]...\\n\"\n+\" test-tool date getnanos [start-nanos]\\n\"\n \" test-tool date is64bit\\n\"\n \" test-tool date time_t-is64bit\\n\";\n \n@@ -82,6 +83,15 @@ static void parse_approx_timestamp(const char **argv, struct timeval *now)\n \t}\n }\n \n+static void getnanos(const char **argv, struct timeval *now)\n+{\n+\tdouble seconds = getnanotime() / 1.0e9;\n+\n+\tif (*argv)\n+\t\tseconds -= strtod(*argv, NULL);\n+\tprintf(\"%lf\\n\", seconds);\n+}\n+\n int cmd__date(int argc, const char **argv)\n {\n \tstruct timeval now;\n@@ -108,6 +118,8 @@ int cmd__date(int argc, const char **argv)\n \t\tparse_approxidate(argv+1, &now);\n \telse if (!strcmp(*argv, \"timestamp\"))\n \t\tparse_approx_timestamp(argv+1, &now);\n+\telse if (!strcmp(*argv, \"getnanos\"))\n+\t\tgetnanos(argv+1, &now);\n \telse if (!strcmp(*argv, \"is64bit\"))\n \t\treturn sizeof(timestamp_t) == 8 ? 0 : 1;\n \telse if (!strcmp(*argv, \"time_t-is64bit\"))\n-- \ngitgitgadget\n\n"},{"id":"367791","messageId":"681f8e65b4d3cf82a5920840aa8dc88bde812964.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 05/21] ci: use a junction on Windows instead of a symlink","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:53Z","receivedAt":"2019-01-27T23:27:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nSymbolic links are still not quite as easy to use on Windows as on Linux\n(for example, on versions older than Windows 10, only administrators can\ncreate symlinks, and on Windows 10 you still need to be in developer\nmode for regular users to have permission), but NTFS junctions can give\nus a way out.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/run-build-and-tests.sh | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 80d72d120f..74d838ea01 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -5,7 +5,10 @@\n \n . ${0%/*}/lib.sh\n \n-ln -s \"$cache_dir/.prove\" t/.prove\n+case \"$CI_OS_NAME\" in\n+windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n+*) ln -s \"$cache_dir/.prove\" t/.prove;;\n+esac\n \n make\n make --quiet test\n-- \ngitgitgadget\n\n"},{"id":"367792","messageId":"7518266a19e4167fbe6459ff54f452573176c296.1548631608.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 18/21] t0061: fix with --with-dashes and RUNTIME_PREFIX","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:27:03Z","receivedAt":"2019-01-27T23:27:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWhen building Git with RUNTIME_PREFIX and starting a test helper from\nt/helper/, it fails to detect the system prefix correctly.\n\nThis is the reason that the warning\n\n\tRUNTIME_PREFIX requested, but prefix computation failed. [...]\n\nto be printed.\n\nIn t0061, we did not expect that to happen, and it actually did not\nhappen in the normal case, because bin-wrappers/test-tool specifically\nsets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool wants\nto know about the runtime prefix).\n\nHowever, with --with-dashes, bin-wrappers/test-tool is no longer called,\nbut t/helper/test-tool is called directly.\n\nSo let's just ignore the RUNTIME_PREFIX warning.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t0061-run-command.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 99a614bc7c..5a2d087bf0 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -166,7 +166,8 @@ test_trace () {\n \texpect=\"$1\"\n \tshift\n \tGIT_TRACE=1 test-tool run-command \"$@\" run-command true 2>&1 >/dev/null | \\\n-\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' >actual &&\n+\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' \\\n+\t\t\t-e '/RUNTIME_PREFIX requested/d' >actual &&\n \techo \"$expect true\" >expect &&\n \ttest_cmp expect actual\n }\n-- \ngitgitgadget\n\n"},{"id":"367793","messageId":"6a8adbe8baedd166f13d4fa74bce712165168f1b.1548631608.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 19/21] tests: optionally skip bin-wrappers/","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:27:04Z","receivedAt":"2019-01-27T23:27:10Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis speeds up the tests by a bit on Windows, where running Unix shell\nscripts (and spawning processes) is not exactly a cheap operation.\n\nSigned-off-by: Johannes Schindelin \n---\n t/README | 9 +++++++++\n t/test-lib.sh | 19 +++++++++++++------\n 2 files changed, 22 insertions(+), 6 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 11ce7675e3..063530234f 100644\n--- a/t/README\n+++ b/t/README\n@@ -170,6 +170,15 @@ appropriately before running \"make\".\n \timplied by other options like --valgrind and\n \tGIT_TEST_INSTALLED.\n \n+--no-bin-wrappers::\n+\tBy default, the test suite uses the wrappers in\n+\t`../bin-wrappers/` to execute `git` and friends. With this option,\n+\t`../git` and friends are run directly. This is not recommended\n+\tin general, as the wrappers contain safeguards to ensure that no\n+\tfiles from an installed Git are used, but can speed up test runs\n+\tespecially on platforms where running shell scripts is expensive\n+\t(most notably, Windows).\n+\n --root=::\n \tCreate \"trash\" directories used to store all temporary data during\n \ttesting under , instead of the t/ directory.\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex c790e98fd2..25e649c997 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -111,6 +111,8 @@ do\n \t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n \t--with-dashes)\n \t\twith_dashes=t ;;\n+\t--no-bin-wrappers)\n+\t\tno_bin_wrappers=t ;;\n \t--no-color)\n \t\tcolor= ;;\n \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n@@ -1214,16 +1216,21 @@ then\n \tPATH=$GIT_TEST_INSTALLED:$GIT_BUILD_DIR/t/helper:$PATH\n \tGIT_EXEC_PATH=${GIT_TEST_EXEC_PATH:-$GIT_EXEC_PATH}\n else # normal case, use ../bin-wrappers only unless $with_dashes:\n-\tgit_bin_dir=\"$GIT_BUILD_DIR/bin-wrappers\"\n-\tif ! test -x \"$git_bin_dir/git\"\n+\tif test -n \"$no_bin_wrappers\"\n \tthen\n-\t\tif test -z \"$with_dashes\"\n+\t\twith_dashes=t\n+\telse\n+\t\tgit_bin_dir=\"$GIT_BUILD_DIR/bin-wrappers\"\n+\t\tif ! test -x \"$git_bin_dir/git\"\n \t\tthen\n-\t\t\tsay \"$git_bin_dir/git is not executable; using GIT_EXEC_PATH\"\n+\t\t\tif test -z \"$with_dashes\"\n+\t\t\tthen\n+\t\t\t\tsay \"$git_bin_dir/git is not executable; using GIT_EXEC_PATH\"\n+\t\t\tfi\n+\t\t\twith_dashes=t\n \t\tfi\n-\t\twith_dashes=t\n+\t\tPATH=\"$git_bin_dir:$PATH\"\n \tfi\n-\tPATH=\"$git_bin_dir:$PATH\"\n \tGIT_EXEC_PATH=$GIT_BUILD_DIR\n \tif test -n \"$with_dashes\"\n \tthen\n-- \ngitgitgadget\n\n"},{"id":"367794","messageId":"9984c6d17bc09fc96c627a7c7fe5ecf83070dffe.1548631608.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 16/21] mingw: try to work around issues with the test cleanup","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:27:02Z","receivedAt":"2019-01-27T23:27:12Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIt seems that every once in a while in the Git for Windows SDK, there\nare some transient file locking issues preventing the test clean up to\ndelete the trash directory. Let's be gentle and try again five seconds\nlater, and only error out if it still fails the second time.\n\nThis change helps Windows, and does not hurt any other platform\n(normally, it is highly unlikely that said deletion fails, and if it\ndoes, normally it will fail again even 5 seconds later).\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex f31a1c8f79..9c0ca5effb 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1104,7 +1104,11 @@ test_done () {\n \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n \n \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n-\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n+\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n+\t\t\t\t# try again in a bit\n+\t\t\t\tsleep 5;\n+\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n+\t\t\t} ||\n \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n \t\tfi\n \t\ttest_at_end_hook_\n-- \ngitgitgadget\n\n"},{"id":"367795","messageId":"dfd3516f39d3b35ed451426d0f87cfbd93ffd06f.1548631608.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 15/21] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:27:01Z","receivedAt":"2019-01-27T23:27:13Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe JUnit XML format lends itself to be presented in a powerful UI,\nwhere you can drill down to the information you are interested in very\nquickly.\n\nFor test failures, this usually means that you want to see the detailed\ntrace of the failing tests.\n\nWith Travis CI, we passed the `--verbose-log` option to get those\ntraces. However, that seems excessive, as we do not need/use the logs in\nalmost all of those cases: only when a test fails do we have a way to\ninclude the trace.\n\nSo let's do something different when using Azure DevOps: let's run all\nthe tests with `--quiet` first, and only if a failure is encountered,\ntry to trace the commands as they are executed.\n\nOf course, we cannot turn on `--verbose-log` after the fact. So let's\njust re-run the test with all the same options, adding `--verbose-log`.\nAnd then munging the output file into the JUnit XML on the fly.\n\nNote: there is an off chance that re-running the test in verbose mode\n\"fixes\" the failures (and this does happen from time to time!). That is\na possibility we should be able to live with. Ideally, we would label\nthis as \"Passed upon rerun\", and Azure Pipelines even know about that\noutcome, but it is not available when using the JUnit XML format for\nnow:\nhttps://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-path-utils.c | 21 +++++++++++++++++++++\n t/test-lib.sh | 22 +++++++++++++++++++++-\n 2 files changed, 42 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex 30211d6d64..6efde6f5ba 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -303,6 +303,27 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn !!res;\n \t}\n \n+\tif (argc == 4 && !strcmp(argv[1], \"skip-n-bytes\")) {\n+\t\tint fd = open(argv[2], O_RDONLY), offset = atoi(argv[3]);\n+\t\tchar buffer[65536];\n+\n+\t\tif (fd < 0)\n+\t\t\tdie_errno(\"could not open '%s'\", argv[2]);\n+\t\tif (lseek(fd, offset, SEEK_SET) < 0)\n+\t\t\tdie_errno(\"could not skip %d bytes\", offset);\n+\t\tfor (;;) {\n+\t\t\tssize_t count = read(fd, buffer, sizeof(buffer));\n+\t\t\tif (count < 0)\n+\t\t\t\tdie_errno(\"could not read '%s'\", argv[2]);\n+\t\t\tif (!count)\n+\t\t\t\tbreak;\n+\t\t\tif (write(1, buffer, count) < 0)\n+\t\t\t\tdie_errno(\"could not write to stdout\");\n+\t\t}\n+\t\tclose(fd);\n+\t\treturn 0;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a3b2166cb5..f31a1c8f79 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -639,8 +639,19 @@ test_failure_ () {\n \t\tjunit_insert=\"\"\n \t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n-\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\t\t\"$(if test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\t\t then\n+\t\t\t\ttest-tool path-utils skip-n-bytes \\\n+\t\t\t\t\t\"$GIT_TEST_TEE_OUTPUT_FILE\" $GIT_TEST_TEE_OFFSET\n+\t\t\t else\n+\t\t\t\tprintf '%s\\n' \"$@\" | sed 1d\n+\t\t\t fi)\")\"\n \t\tjunit_insert=\"$junit_insert\"\n+\t\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\tthen\n+\t\t\tjunit_insert=\"$junit_insert$(xml_attr_encode \\\n+\t\t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n+\t\tfi\n \t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n \tfi\n \ttest_failure=$(($test_failure + 1))\n@@ -931,6 +942,11 @@ test_finish_ () {\n \techo >&3 \"\"\n \tmaybe_teardown_valgrind\n \tmaybe_teardown_verbose\n+\tif test -n \"$GIT_TEST_TEE_OFFSET\"\n+\tthen\n+\t\tGIT_TEST_TEE_OFFSET=$(test-tool path-utils file-size \\\n+\t\t\t\"$GIT_TEST_TEE_OUTPUT_FILE\")\n+\tfi\n }\n \n test_skip () {\n@@ -1280,6 +1296,10 @@ then\n \t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n \twrite_junit_xml --truncate \"\" \" \"\n \tjunit_suite_start=$(test-tool date getnanos)\n+\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\tthen\n+\t\tGIT_TEST_TEE_OFFSET=0\n+\tfi\n fi\n \n # Provide an implementation of the 'yes' utility\n-- \ngitgitgadget\n\n"},{"id":"367796","messageId":"4303bbd4e816a110c18ed29e8ec371ae50afaeba.1548631608.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 14/21] tests: avoid calling Perl just to determine file sizes","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:27:00Z","receivedAt":"2019-01-27T23:27:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIt is a bit ridiculous to spin up a full-blown Perl instance (especially\non Windows, where that means spinning up a full POSIX emulation layer,\nAKA the MSYS2 runtime) just to tell how large a given file is.\n\nSo let's just use the test-tool to do that job instead.\n\nThis command will also be used over the next commits, to allow for\ncutting out individual test cases' verbose log from the file generated\nvia --verbose-log.\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-path-utils.c | 12 ++++++++++++\n t/t0021-conversion.sh | 2 +-\n t/t1050-large.sh | 2 +-\n t/t5315-pack-objects-compression.sh | 2 +-\n t/t9303-fast-import-compression.sh | 2 +-\n 5 files changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex ae091d9b3e..30211d6d64 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -291,6 +291,18 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn !!res;\n \t}\n \n+\tif (argc > 2 && !strcmp(argv[1], \"file-size\")) {\n+\t\tint res = 0, i;\n+\t\tstruct stat st;\n+\n+\t\tfor (i = 2; i < argc; i++)\n+\t\t\tif (stat(argv[i], &st))\n+\t\t\t\tres = error_errno(\"Cannot stat '%s'\", argv[i]);\n+\t\t\telse\n+\t\t\t\tprintf(\"%\"PRIuMAX\"\\n\", (uintmax_t)st.st_size);\n+\t\treturn !!res;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex fd5f1ac649..e10f5f787f 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -24,7 +24,7 @@ generate_random_characters () {\n }\n \n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n filter_git () {\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 1a9b21b293..dcb4dbba67 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -8,7 +8,7 @@ test_description='adding and checking out large blobs'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n test_expect_success setup '\ndiff --git a/t/t5315-pack-objects-compression.sh b/t/t5315-pack-objects-compression.sh\nindex 34c47dae09..df970d7584 100755\n--- a/t/t5315-pack-objects-compression.sh\n+++ b/t/t5315-pack-objects-compression.sh\n@@ -7,7 +7,7 @@ test_description='pack-object compression configuration'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n test_expect_success setup '\ndiff --git a/t/t9303-fast-import-compression.sh b/t/t9303-fast-import-compression.sh\nindex 856219f46a..5045f02a53 100755\n--- a/t/t9303-fast-import-compression.sh\n+++ b/t/t9303-fast-import-compression.sh\n@@ -6,7 +6,7 @@ test_description='compression setting of fast-import utility'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n import_large () {\n-- \ngitgitgadget\n\n"},{"id":"367797","messageId":"eda46863e85f9b03a983697102cb4716c614f3ee.1548631608.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 13/21] README: add a build badge (status of the Azure Pipelines build)","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:59Z","receivedAt":"2019-01-27T23:27:15Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nJust like so many other OSS projects, we now also have a build badge.\n\nSigned-off-by: Johannes Schindelin \n---\n README.md | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/README.md b/README.md\nindex f920a42fad..764c480c66 100644\n--- a/README.md\n+++ b/README.md\n@@ -1,3 +1,5 @@\n+[![Build Status](https://dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2)\n+\n Git - fast, scalable, distributed revision control system\n =========================================================\n \n-- \ngitgitgadget\n\n"},{"id":"367798","messageId":"1cef14e4c04e27751445397538bc824154a9d42e.1548631608.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 12/21] mingw: be more generous when wrapping up the setitimer() emulation","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:58Z","receivedAt":"2019-01-27T23:27:17Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nEvery once in a while, the Azure Pipeline fails with some semi-random\n\n\terror: timer thread did not terminate timely\n\nThis error message means that the thread that is used to emulate the\nsetitimer() function did not terminate within 1,000 milliseconds.\n\nThe most likely explanation (and therefore the one we should assume to\nbe true, according to Occam's Razor) is that the timeout of one second\nis simply not enough because we try to run so many tasks in parallel.\n\nSo let's give it ten seconds instead of only one. That should be enough.\n\nSigned-off-by: Johannes Schindelin \n---\n compat/mingw.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex b459e1a291..e0dfe8844d 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -2100,7 +2100,7 @@ static void stop_timer_thread(void)\n \tif (timer_event)\n \t\tSetEvent(timer_event);\t/* tell thread to terminate */\n \tif (timer_thread) {\n-\t\tint rc = WaitForSingleObject(timer_thread, 1000);\n+\t\tint rc = WaitForSingleObject(timer_thread, 10000);\n \t\tif (rc == WAIT_TIMEOUT)\n \t\t\terror(\"timer thread did not terminate timely\");\n \t\telse if (rc != WAIT_OBJECT_0)\n-- \ngitgitgadget\n\n"},{"id":"367800","messageId":"52d63018c9d4ef0374fe0dca58fc1f8a856736c9.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 11/21] ci: use git-sdk-64-minimal build artifact","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:57Z","receivedAt":"2019-01-27T23:27:18Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nInstead of a shallow fetch followed by a sparse checkout, we are\nbetter off by using a separate, dedicated Pipeline that bundles\nthe SDK as a build artifact, and then consuming that build artifact\nhere.\n\nIn fact, since this artifact will be used a lot, we spent substantial\ntime on figuring out a minimal subset of the Git for Windows SDK, just\nenough to build and test Git. The result is a size reduction from around\n1GB (compressed) to around 55MB (compressed). This also comes with the\nchange where we now call `usr\\bin\\bash.exe` directly, as `git-cmd.exe`\nis not included in the minimal SDK.\n\nThat reduces the time to initialize Git for Windows' SDK from anywhere\nbetween 2m30s-7m to a little over 1m.\n\nNote: in theory, we could also use the DownloadBuildArtifacts@0 task\nhere. However, restricted permissions that are in effect when building\nfrom forks would let this fail for PR builds, defeating the whole\npurpose of the Azure Pipelines support for git.git.\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 44 +++++++++-----------------------------------\n 1 file changed, 9 insertions(+), 35 deletions(-)\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex 0f7b2125a1..480e841a85 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -18,44 +18,18 @@ jobs:\n env:\n GITFILESHAREPWD: $(gitfileshare.pwd)\n - powershell: |\n- # Helper to check the error level of the latest command (exit with error when appropriate)\n- function c() { if (!$?) { exit(1) } }\n-\n- # Add build agent's MinGit to PATH\n- $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n-\n- # Helper to initialize (or update) a Git worktree\n- function init ($path, $url, $set_origin) {\n- if (Test-Path $path) {\n- cd $path; c\n- if (Test-Path .git) {\n- & git init; c\n- } else {\n- & git status\n- }\n- } else {\n- & git init $path; c\n- cd $path; c\n- }\n- & git config core.autocrlf false; c\n- & git config core.untrackedCache true; c\n- if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n- & git remote add origin $url; c\n- }\n- & git fetch --depth=1 $url master; c\n- & git reset --hard FETCH_HEAD; c\n- & git clean -df; c\n- }\n-\n- # Initialize Git for Windows' SDK\n- $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n- init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+ $urlbase = \"https://dev.azure.com/git-for-windows/git/_apis/build/builds\"\n+ $id = ((Invoke-WebRequest -UseBasicParsing \"${urlbase}?definitions=22&statusFilter=completed&resultFilter=succeeded&`$top=1\").content | ConvertFrom-JSON).value[0].id\n+ $downloadUrl = ((Invoke-WebRequest -UseBasicParsing \"${urlbase}/$id/artifacts\").content | ConvertFrom-JSON).value[1].resource.downloadUrl\n+ (New-Object Net.WebClient).DownloadFile($downloadUrl,\"git-sdk-64-minimal.zip\")\n+ Expand-Archive git-sdk-64-minimal.zip -DestinationPath . -Force\n+ Remove-Item git-sdk-64-minimal.zip\n \n # Let Git ignore the SDK and the test-cache\n- \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n- displayName: 'Initialize the Git for Windows SDK'\n+ \"/git-sdk-64-minimal/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n+ displayName: 'Download git-sdk-64-minimal'\n - powershell: |\n- & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n export DEVELOPER=1\n export NO_PERL=1\n export NO_SVN_TESTS=1\n-- \ngitgitgadget\n\n"},{"id":"367799","messageId":"6044b919b0c186aa1f20b56b4235696a248983cf.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 09/21] Add a build definition for Azure DevOps","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:56Z","receivedAt":"2019-01-27T23:27:19Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis commit adds an azure-pipelines.yml file which is Azure DevOps'\nequivalent to Travis CI's .travis.yml.\n\nThe main idea is to replicate the Travis configuration as faithfully as\npossible, to make it easy to compare the Azure Pipeline builds to the\nTravis ones (spoiler: some parts, especially the macOS jobs, are way\nfaster in Azure Pileines). Meaning: the number and the order of the jobs\nadded in this commit faithfully replicates what we have in .travis.yml.\n\nNote: Our .travis.yml configuration has a Windows part that is *not*\nreplicated in the Azure Pipelines definition. The reason is easy to see:\nAs Travis cannot support our Windws needs (even with the preliminary\nWindows support that was recently added to Travis after waiting for\n*years* for that feature, our test suite would simply hit Travis'\ntimeout every single time).\n\nTo make things a bit easier to understand, we refrain from using the\n`matrix` feature here because (while it is powerful) it can be a bit\nconfusing to users who are not familiar with CI setups. Therefore, we\nuse a separate phase even for similar configurations (such as GCC vs\nClang on Linux, GCC vs Clang on macOS).\n\nAlso, we make use of the shiny new feature we just introduced where the\ntest suite can output JUnit-style .xml files. This information is made\navailable in a nice UI that allows the viewer to filter by phase and/or\ntest number, and to see trends such as: number of (failing) tests, time\nspent running the test suite, etc. (While this seemingly contradicts the\nintention to replicate the Travis configuration as faithfully as\npossible, it is just too nice to show off that capability here already.)\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 259 ++++++++++++++++++++++++++++++++++++++++++\n ci/mount-fileshare.sh | 25 ++++\n 2 files changed, 284 insertions(+)\n create mode 100644 azure-pipelines.yml\n create mode 100755 ci/mount-fileshare.sh\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nnew file mode 100644\nindex 0000000000..8cdef105c6\n--- /dev/null\n+++ b/azure-pipelines.yml\n@@ -0,0 +1,259 @@\n+resources:\n+- repo: self\n+ fetchDepth: 1\n+\n+jobs:\n+- job: linux_clang\n+ displayName: linux-clang\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin &&\n+\n+ export CC=clang || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-clang'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: linux_gcc\n+ displayName: linux-gcc\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo add-apt-repository ppa:ubuntu-toolchain-r/test &&\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2 language-pack-is git-svn gcc-8 || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-gcc'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: osx_clang\n+ displayName: osx-clang\n+ condition: succeeded()\n+ pool: Hosted macOS\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ export CC=clang\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-clang'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: osx_gcc\n+ displayName: osx-gcc\n+ condition: succeeded()\n+ pool: Hosted macOS\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-gcc'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: gettext_poison\n+ displayName: GETTEXT_POISON\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev &&\n+\n+ export jobname=GETTEXT_POISON || exit 1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'gettext-poison'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: linux32\n+ displayName: Linux32\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ res=0\n+ sudo AGENT_OS=\"$AGENT_OS\" BUILD_BUILDNUMBER=\"$BUILD_BUILDNUMBER\" BUILD_REPOSITORY_URI=\"$BUILD_REPOSITORY_URI\" BUILD_SOURCEBRANCH=\"$BUILD_SOURCEBRANCH\" BUILD_SOURCEVERSION=\"$BUILD_SOURCEVERSION\" SYSTEM_PHASENAME=\"$SYSTEM_PHASENAME\" SYSTEM_TASKDEFINITIONSURI=\"$SYSTEM_TASKDEFINITIONSURI\" SYSTEM_TEAMPROJECT=\"$SYSTEM_TEAMPROJECT\" CC=$CC MAKEFLAGS=\"$MAKEFLAGS\" bash -lxc ci/run-linux32-docker.sh || res=1\n+\n+ sudo chmod a+r t/out/TEST-*.xml\n+ test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || res=1\n+ exit $res\n+ displayName: 'ci/run-linux32-docker.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux32'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: static_analysis\n+ displayName: StaticAnalysis\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get install -y coccinelle &&\n+\n+ export jobname=StaticAnalysis &&\n+\n+ ci/run-static-analysis.sh || exit 1\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-static-analysis.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+\n+- job: documentation\n+ displayName: Documentation\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get install -y asciidoc xmlto asciidoctor &&\n+\n+ export ALREADY_HAVE_ASCIIDOCTOR=yes. &&\n+ export jobname=Documentation &&\n+\n+ ci/test-documentation.sh || exit 1\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/test-documentation.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\ndiff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\nnew file mode 100755\nindex 0000000000..26b58a8096\n--- /dev/null\n+++ b/ci/mount-fileshare.sh\n@@ -0,0 +1,25 @@\n+#!/bin/sh\n+\n+die () {\n+\techo \"$*\" >&2\n+\texit 1\n+}\n+\n+test $# = 4 ||\n+die \"Usage: $0 \"\n+\n+mkdir -p \"$4\" || die \"Could not create $4\"\n+\n+case \"$(uname -s)\" in\n+Linux)\n+\tsudo mount -t cifs -o vers=3.0,username=\"$2\",password=\"$3\",dir_mode=0777,file_mode=0777,serverino \"$1\" \"$4\"\n+\t;;\n+Darwin)\n+\tpass=\"$(echo \"$3\" | sed -e 's/\\//%2F/g' -e 's/+/%2B/g')\" &&\n+\tmount -t smbfs,soft \"smb://$2:$pass@${1#//}\" \"$4\"\n+\t;;\n+*)\n+\tdie \"No support for $(uname -s)\"\n+\t;;\n+esac ||\n+die \"Could not mount $4\"\n-- \ngitgitgadget\n\n"},{"id":"367801","messageId":"6ff87d17729db3dca73e74162cd8f148ce63efdd.1548631608.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 21/21] ci: parallelize testing on Windows","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:27:06Z","receivedAt":"2019-01-27T23:27:22Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe fact that Git's test suite is implemented in Unix shell script that\nis as portable as we can muster, combined with the fact that Unix shell\nscripting is foreign to Windows (and therefore has to be emulated),\nresults in pretty abysmal speed of the test suite on that platform, for\npretty much no other reason than that language choice.\n\nFor comparison: while the Linux build & test is typically done within\nabout 8 minutes, the Windows build & test typically lasts about 80\nminutes in Azure Pipelines.\n\nTo help with that, let's use the Azure Pipeline feature where you can\nparallelize jobs, make jobs depend on each other, and pass artifacts\nbetween them.\n\nThe tests are distributed using the following heuristic: listing all\ntest scripts ordered by size in descending order (as a cheap way to\nestimate the overall run time), every Nth script is run (where N is the\ntotal number of parallel jobs), starting at the index corresponding to\nthe parallel job. This slicing is performed by a new function that is\nadded to the `test-tool`.\n\nTo optimize the overall runtime of the entire Pipeline, we need to move\nthe Windows jobs to the beginning (otherwise there would be a very\ndecent chance for the Pipeline to be run only the Windows build, while\nall the parallel Windows test jobs wait for this single one).\n\nWe use Azure Pipelines Artifacts for both the minimal Git for Windows\nSDK as well as the built executables, as deduplication and caching close\nto the agents makes that really fast. For comparison: while downloading\nand unpacking the minimal Git for Windows SDK via PowerShell takes only\none minute (down from anywhere between 2.5 to 7 when using a shallow\nclone), uploading it as Pipeline Artifact takes less than 30s and\ndownloading and unpacking less than 20s (sometimes even as little as\nonly twelve seconds).\n\nSigned-off-by: Johannes Schindelin \n---\n Makefile | 10 +++++\n azure-pipelines.yml | 79 ++++++++++++++++++++++++++++++++++----\n ci/make-test-artifacts.sh | 12 ++++++\n ci/run-test-slice.sh | 17 ++++++++\n t/helper/test-path-utils.c | 31 +++++++++++++++\n 5 files changed, 141 insertions(+), 8 deletions(-)\n create mode 100755 ci/make-test-artifacts.sh\n create mode 100755 ci/run-test-slice.sh\n\ndiff --git a/Makefile b/Makefile\nindex 044b4f77bd..daa318fe17 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2927,6 +2927,16 @@ rpm::\n \t@false\n .PHONY: rpm\n \n+artifacts-tar:: $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) \\\n+\t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n+\t\t$(NO_INSTALL) $(MOFILES)\n+\t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n+\t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n+\ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n+\tmkdir -p \"$(ARTIFACTS_DIRECTORY)\"\n+\t$(TAR) czf \"$(ARTIFACTS_DIRECTORY)/artifacts.tar.gz\" $^ templates/blt/\n+.PHONY: artifacts-tar\n+\n htmldocs = git-htmldocs-$(GIT_VERSION)\n manpages = git-manpages-$(GIT_VERSION)\n .PHONY: dist-doc distclean\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex 480e841a85..c329b7218b 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -3,8 +3,8 @@ resources:\n fetchDepth: 1\n \n jobs:\n-- job: windows\n- displayName: Windows\n+- job: windows_build\n+ displayName: Windows Build\n condition: succeeded()\n pool: Hosted\n timeoutInMinutes: 240\n@@ -30,21 +30,84 @@ jobs:\n displayName: 'Download git-sdk-64-minimal'\n - powershell: |\n & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n- export DEVELOPER=1\n- export NO_PERL=1\n- export NO_SVN_TESTS=1\n- export GIT_TEST_SKIP_REBASE_P=1\n+ ci/make-test-artifacts.sh artifacts\n+ \"@\n+ if (!$?) { exit(1) }\n+ displayName: Build\n+ env:\n+ HOME: $(Build.SourcesDirectory)\n+ MSYSTEM: MINGW64\n+ DEVELOPER: 1\n+ NO_PERL: 1\n+ - task: PublishPipelineArtifact@0\n+ displayName: 'Publish Pipeline Artifact: test artifacts'\n+ inputs:\n+ artifactName: 'windows-artifacts'\n+ targetPath: '$(Build.SourcesDirectory)\\artifacts'\n+ - task: PublishPipelineArtifact@0\n+ displayName: 'Publish Pipeline Artifact: git-sdk-64-minimal'\n+ inputs:\n+ artifactName: 'git-sdk-64-minimal'\n+ targetPath: '$(Build.SourcesDirectory)\\git-sdk-64-minimal'\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'Unmount test-cache'\n+ condition: true\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+\n+- job: windows_test\n+ displayName: Windows Test\n+ dependsOn: windows_build\n+ condition: succeeded()\n+ pool: Hosted\n+ timeoutInMinutes: 240\n+ strategy:\n+ parallel: 10\n+ steps:\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n+ }\n+ displayName: 'Mount test-cache'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: DownloadPipelineArtifact@0\n+ displayName: 'Download Pipeline Artifact: test artifacts'\n+ inputs:\n+ artifactName: 'windows-artifacts'\n+ targetPath: '$(Build.SourcesDirectory)'\n+ - task: DownloadPipelineArtifact@0\n+ displayName: 'Download Pipeline Artifact: git-sdk-64-minimal'\n+ inputs:\n+ artifactName: 'git-sdk-64-minimal'\n+ targetPath: '$(Build.SourcesDirectory)\\git-sdk-64-minimal'\n+ - powershell: |\n+ & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n+ test -f artifacts.tar.gz || {\n+ echo No test artifacts found\\; skipping >&2\n+ exit 0\n+ }\n+ tar xf artifacts.tar.gz || exit 1\n+\n+ # Let Git ignore the SDK and the test-cache\n+ printf '%s\\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude\n \n- ci/run-build-and-tests.sh || {\n+ ci/run-test-slice.sh `$SYSTEM_JOBPOSITIONINPHASE `$SYSTEM_TOTALJOBSINPHASE || {\n ci/print-test-failures.sh\n exit 1\n }\n \"@\n if (!$?) { exit(1) }\n- displayName: 'Build & Test'\n+ displayName: 'Test (parallel)'\n env:\n HOME: $(Build.SourcesDirectory)\n MSYSTEM: MINGW64\n+ NO_SVN_TESTS: 1\n+ GIT_TEST_SKIP_REBASE_P: 1\n - powershell: |\n if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\ndiff --git a/ci/make-test-artifacts.sh b/ci/make-test-artifacts.sh\nnew file mode 100755\nindex 0000000000..646967481f\n--- /dev/null\n+++ b/ci/make-test-artifacts.sh\n@@ -0,0 +1,12 @@\n+#!/bin/sh\n+#\n+# Build Git and store artifacts for testing\n+#\n+\n+mkdir -p \"$1\" # in case ci/lib.sh decides to quit early\n+\n+. ${0%/*}/lib.sh\n+\n+make artifacts-tar ARTIFACTS_DIRECTORY=\"$1\"\n+\n+check_unignored_build_artifacts\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nnew file mode 100755\nindex 0000000000..f8c2c3106a\n--- /dev/null\n+++ b/ci/run-test-slice.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+#\n+# Test Git in parallel\n+#\n+\n+. ${0%/*}/lib.sh\n+\n+case \"$CI_OS_NAME\" in\n+windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n+*) ln -s \"$cache_dir/.prove\" t/.prove;;\n+esac\n+\n+make --quiet -C t T=\"$(cd t &&\n+\t./helper/test-tool path-utils slice-tests \"$1\" \"$2\" t[0-9]*.sh |\n+\ttr '\\n' ' ')\"\n+\n+check_unignored_build_artifacts\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex 6efde6f5ba..5d543ad21f 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -177,6 +177,14 @@ static int is_dotgitmodules(const char *path)\n \treturn is_hfs_dotgitmodules(path) || is_ntfs_dotgitmodules(path);\n }\n \n+static int cmp_by_st_size(const void *a, const void *b)\n+{\n+\tintptr_t x = (intptr_t)((struct string_list_item *)a)->util;\n+\tintptr_t y = (intptr_t)((struct string_list_item *)b)->util;\n+\n+\treturn x > y ? -1 : (x < y ? +1 : 0);\n+}\n+\n int cmd__path_utils(int argc, const char **argv)\n {\n \tif (argc == 3 && !strcmp(argv[1], \"normalize_path_copy\")) {\n@@ -324,6 +332,29 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn 0;\n \t}\n \n+\tif (argc > 5 && !strcmp(argv[1], \"slice-tests\")) {\n+\t\tint res = 0;\n+\t\tlong offset, stride, i;\n+\t\tstruct string_list list = STRING_LIST_INIT_NODUP;\n+\t\tstruct stat st;\n+\n+\t\toffset = strtol(argv[2], NULL, 10);\n+\t\tstride = strtol(argv[3], NULL, 10);\n+\t\tif (stride < 1)\n+\t\t\tstride = 1;\n+\t\tfor (i = 4; i < argc; i++)\n+\t\t\tif (stat(argv[i], &st))\n+\t\t\t\tres = error_errno(\"Cannot stat '%s'\", argv[i]);\n+\t\t\telse\n+\t\t\t\tstring_list_append(&list, argv[i])->util =\n+\t\t\t\t\t(void *)(intptr_t)st.st_size;\n+\t\tQSORT(list.items, list.nr, cmp_by_st_size);\n+\t\tfor (i = offset; i < list.nr; i+= stride)\n+\t\t\tprintf(\"%s\\n\", list.items[i].string);\n+\n+\t\treturn !!res;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\n-- \ngitgitgadget\n"},{"id":"367802","messageId":"a8101ae3e18dfad3bbdb6ae5dce798ba934a053e.1548631608.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 17/21] tests: add t/helper/ to the PATH with --with-dashes","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:27:03Z","receivedAt":"2019-01-27T23:27:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWe really need to be able to find the test helpers... Really. This\nchange was forgotten when we moved the test helpers into t/helper/\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9c0ca5effb..c790e98fd2 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1227,7 +1227,7 @@ else # normal case, use ../bin-wrappers only unless $with_dashes:\n \tGIT_EXEC_PATH=$GIT_BUILD_DIR\n \tif test -n \"$with_dashes\"\n \tthen\n-\t\tPATH=\"$GIT_BUILD_DIR:$PATH\"\n+\t\tPATH=\"$GIT_BUILD_DIR:$GIT_BUILD_DIR/t/helper:$PATH\"\n \tfi\n fi\n GIT_TEMPLATE_DIR=\"$GIT_BUILD_DIR\"/templates/blt\n-- \ngitgitgadget\n\n"},{"id":"367803","messageId":"b39e16527da862c64e4338cf3e3fea00345f3a81.1548631608.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 20/21] ci: speed up Windows phase","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:27:05Z","receivedAt":"2019-01-27T23:27:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nAs Unix shell scripting comes at a hefty price on Windows, we have to\nsee where we can save some time to run the test suite.\n\nLet's skip the chain linting and the bin-wrappers/ redirection on\nWindows; this seems to shave of anywhere between 10-30% from the overall\nruntime.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 5505776876..c2bc6c68b9 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -127,6 +127,8 @@ then\n \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n \texport MAKEFLAGS=\"--jobs=10\"\n+\ttest windows_nt != \"$CI_OS_NAME\" ||\n+\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n else\n \techo \"Could not identify CI type\" >&2\n \texit 1\n-- \ngitgitgadget\n\n"},{"id":"367804","messageId":"3a2993c826904a92f5a3198e8a5764991a7d430c.1548631607.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v5 10/21] ci: add a Windows job to the Azure Pipelines definition","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-27T23:26:57Z","receivedAt":"2019-01-27T23:27:26Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nPreviously, we did not have robust support for Windows in our CI\ndefinition, simply because Travis cannot accommodate our needs (even\nafter Travis added experimental Windows support very recently, it takes\nlonger than Travis' 50 minute timeout to build Git and run the test\nsuite on Windows). Instead, we used a hack that started a dedicated\nAzure Pipeline from Travis and waited for the output, often timing out\n(which is quite fragile, as we found out).\n\nWith this commit, we finally have first-class support for Windows in our\nCI definition (in the Azure Pipelines one, that is).\n\nDue to our reliance on Unix shell scripting in the test suite, combined\nwith the challenges on executing such scripts on Windows, the Windows\njob currently takes a whopping ~1h20m to complete. Which is *far* longer\nthan the next-longest job takes (linux-gcc, ~35m).\n\nNow, Azure Pipelines's free tier for open source projects (such as Git)\noffers up to 10 concurrent jobs for free, meaning that the overall run\ntime will be dominated by the slowest job(s).\n\nTherefore, it makes sense to start the Windows job first, to minimize\nthe time the entire build takes from start to end (which is now pretty\nsafely the run time of the Windows job).\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 91 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 91 insertions(+)\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex 8cdef105c6..0f7b2125a1 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -3,6 +3,97 @@ resources:\n fetchDepth: 1\n \n jobs:\n+- job: windows\n+ displayName: Windows\n+ condition: succeeded()\n+ pool: Hosted\n+ timeoutInMinutes: 240\n+ steps:\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n+ }\n+ displayName: 'Mount test-cache'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - powershell: |\n+ # Helper to check the error level of the latest command (exit with error when appropriate)\n+ function c() { if (!$?) { exit(1) } }\n+\n+ # Add build agent's MinGit to PATH\n+ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n+\n+ # Helper to initialize (or update) a Git worktree\n+ function init ($path, $url, $set_origin) {\n+ if (Test-Path $path) {\n+ cd $path; c\n+ if (Test-Path .git) {\n+ & git init; c\n+ } else {\n+ & git status\n+ }\n+ } else {\n+ & git init $path; c\n+ cd $path; c\n+ }\n+ & git config core.autocrlf false; c\n+ & git config core.untrackedCache true; c\n+ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n+ & git remote add origin $url; c\n+ }\n+ & git fetch --depth=1 $url master; c\n+ & git reset --hard FETCH_HEAD; c\n+ & git clean -df; c\n+ }\n+\n+ # Initialize Git for Windows' SDK\n+ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n+ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+\n+ # Let Git ignore the SDK and the test-cache\n+ \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n+ displayName: 'Initialize the Git for Windows SDK'\n+ - powershell: |\n+ & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ export DEVELOPER=1\n+ export NO_PERL=1\n+ export NO_SVN_TESTS=1\n+ export GIT_TEST_SKIP_REBASE_P=1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+ \"@\n+ if (!$?) { exit(1) }\n+ displayName: 'Build & Test'\n+ env:\n+ HOME: $(Build.SourcesDirectory)\n+ MSYSTEM: MINGW64\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'Unmount test-cache'\n+ condition: true\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'windows'\n+ platform: Windows\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n - job: linux_clang\n displayName: linux-clang\n condition: succeeded()\n-- \ngitgitgadget\n\n"},{"id":"367811","messageId":"xmqq36pdcz3o.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"d613c79aff9c65ab9c3723a876c99e6f35a2f133.1548254412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 18/21] t0061: fix with --with-dashes and RUNTIME_PREFIX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-28T02:09:15Z","receivedAt":"2019-01-28T02:09:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Johannes Schindelin via GitGitGadget\" \nwrites:\n\n> From: Johannes Schindelin \n>\n> When building Git with RUNTIME_PREFIX and starting a test helper from\n> t/helper/, it fails to detect the system prefix correctly.\n>\n> This is the reason that the warning\n>\n> \tRUNTIME_PREFIX requested, but prefix computation failed. [...]\n>\n> to be printed.\n>\n> In t0061, we did not expect that to happen, and it actually did not\n> happen in the normal case, because bin-wrappers/test-tool specifically\n> sets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool wants\n> to know about the runtime prefix).\n>\n> However, with --with-dashes, bin-wrappers/test-tool is no longer called,\n> but t/helper/test-tool is called directly.\n>\n> So let's just ignore the RUNTIME_PREFIX warning.\n\nTwo questions that would come to the readers' minds are\n\n - Why \"it fails to detect the system prefix correctly\"? Is that a\n bug waiting to hurt end users?\n\n - Why is it better not to bother fixing that failure? Is it\n because this happens only in the test helper and won't hurt end\n users?\n\nI do not mind this particular \"sweeping it under the rug\" if the\nrationale is \"it only is the strange set-up of test-tool that causes\nit, and we shouldn't burden the code shared with the actual runtime\nto compute runtime prefix just to fix this bug\". I do think it is\nnot productive to bend backwards if it is only done in order to work\naround unusual setup in the test helper binary and I do agree that\nignoring the warning is the right solution.\n\nI just do not like to see that our commits do not explain why we\nchose to ignore, instead of \"fix\", the issue.\n\n> Signed-off-by: Johannes Schindelin \n> ---\n> t/t0061-run-command.sh | 3 ++-\n> 1 file changed, 2 insertions(+), 1 deletion(-)\n>\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index 99a614bc7c..5a2d087bf0 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -166,7 +166,8 @@ test_trace () {\n> \texpect=\"$1\"\n> \tshift\n> \tGIT_TRACE=1 test-tool run-command \"$@\" run-command true 2>&1 >/dev/null | \\\n> -\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' >actual &&\n> +\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' \\\n> +\t\t\t-e '/RUNTIME_PREFIX requested/d' >actual &&\n> \techo \"$expect true\" >expect &&\n> \ttest_cmp expect actual\n> }\n"},{"id":"367831","messageId":"nycvar.QRO.7.76.6.1901281701030.41@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"20190125132545.GD6702@szeder.dev","subject":"Re: [PATCH v4 04/21] ci: inherit --jobs via MAKEFLAGS in run-build-and-tests","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-28T16:02:09Z","receivedAt":"2019-01-28T16:02:35Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Gábor,\n\nOn Fri, 25 Jan 2019, SZEDER Gábor wrote:\n\n> On Wed, Jan 23, 2019 at 02:22:10PM -0800, Junio C Hamano wrote:\n> > \"Johannes Schindelin via GitGitGadget\" \n> > writes:\n> > \n> > > From: Johannes Schindelin \n> > >\n> > > Let's not decide in the generic ci/ script how many jobs to run in\n> > > parallel; it is easy enough to hand that information down via the\n> > > `MAKEFLAGS`.\n> > >\n> > > Signed-off-by: Johannes Schindelin \n> > > ---\n> > > ci/run-build-and-tests.sh | 2 +-\n> > > 1 file changed, 1 insertion(+), 1 deletion(-)\n> > >\n> > > diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\n> > > index db342bb6a8..80d72d120f 100755\n> > > --- a/ci/run-build-and-tests.sh\n> > > +++ b/ci/run-build-and-tests.sh\n> > > @@ -7,7 +7,7 @@\n> > > \n> > > ln -s \"$cache_dir/.prove\" t/.prove\n> > > \n> > > -make --jobs=2\n> > > +make\n> > > make --quiet test\n> > > if test \"$jobname\" = \"linux-gcc\"\n> > > then\n> > \n> > As there is no assignment to MAKEFLAGS in this patch, is it intended\n> > for this step to change behaviour (possibly with the intention to\n> > add \"default 2 jobs at least under travis\" back later in the\n> > series)? Not that it matters too much, but it is unnerving to see\n> > that the proposed log message promising \"it is easy enough\" while\n> > not actually doing so, without expressing an intention.\n> \n> Furthermore, there are several other 'ci/run-.sh' scripts\n> that still run 'make -j N'.\n\nIndeed. I removed those `--jobs=2` options from those.\n\nGranted, I did not audit whether there were `make` calls that did not have\nany `-j` option, but I think it would be safe to parallelize all of them\nvia `MAKEFLAGS`.\n\nCiao,\nDscho"},{"id":"367857","messageId":"xmqqva28aa7v.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"xmqqsgxdelqy.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 06/21] test-date: add a subcommand to measure times in shell scripts","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-28T18:49:40Z","receivedAt":"2019-01-28T18:49:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Junio C Hamano writes:\n>\n>> I think the goal to have our own stopwatch so that we do not have to\n>> worry about differences among system-provided ones makes sense.\n>>\n>> The only thing that may become an issue is how widely available\n>> getnanotime() is. As \"test-date\" itself is built on any platform an\n>> end-user/developer runs our tests, which is wider set of platforms\n>> than what we run Travis and other CIs on, unconditionally relying on\n>> its availability might pose an issue.\n>\n> Sorry for a false alarm, as the codebase in many places like\n> fsmonitor, progress, trace and wt-status have been assuming\n> getnanotime() to be available for quite some time, and this is just\n> another user of the same function.\n\nAnd there was yet another misunderstanding on my part. I thought\nthat the mention of getnanotime() was about\n\nhttps://www.freebsd.org/cgi/man.cgi?query=getnanotime&sektion=9\n\nand I did not realize that the hits I saw in \"git grep getnanotime\"\nwere referring to an unrelated function trace.c::getnanotime() of\nour own. Of course, it is safe to use that function in the tests\n;-)\n\nAgain, sorry for the false alarm.\n\nWe _might_ get a complaint from freebsd devs when they want to use\ntheir getnanotime(9) to implement highres_nanos(), and the cleanest\nsolution to that complaint will be to rename our own getnanotime()\nto git_nanotime() or something, but (1) that is totally outside the\nscope of this series that adds one more caller to the function and\n(2) we can do that when we actually get such a complaint.\n\n\n"},{"id":"367898","messageId":"nycvar.QRO.7.76.6.1901282337550.41@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqq36pdcz3o.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 18/21] t0061: fix with --with-dashes and RUNTIME_PREFIX","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-28T22:44:52Z","receivedAt":"2019-01-28T22:45:16Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Sun, 27 Jan 2019, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" \n> writes:\n> \n> > From: Johannes Schindelin \n> >\n> > When building Git with RUNTIME_PREFIX and starting a test helper from\n> > t/helper/, it fails to detect the system prefix correctly.\n> >\n> > This is the reason that the warning\n> >\n> > \tRUNTIME_PREFIX requested, but prefix computation failed. [...]\n> >\n> > to be printed.\n> >\n> > In t0061, we did not expect that to happen, and it actually did not\n> > happen in the normal case, because bin-wrappers/test-tool specifically\n> > sets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool wants\n> > to know about the runtime prefix).\n> >\n> > However, with --with-dashes, bin-wrappers/test-tool is no longer called,\n> > but t/helper/test-tool is called directly.\n> >\n> > So let's just ignore the RUNTIME_PREFIX warning.\n> \n> Two questions that would come to the readers' minds are\n> \n> - Why \"it fails to detect the system prefix correctly\"? Is that a\n> bug waiting to hurt end users?\n\nI recall explaining that already in a different thread (one that actually\n*was* about the RUNTIME_PREFIX feature, where you asked why we cannot test\nfor it in the test suite): to verify that we are in a valid Git\ninstallation location, we verify that the `git` executable is either in\n`/bin` or in `/libexec/git-core` (we need to check that,\nas the prefix is the parent directory in one case, and the grand parent\ndirectory in the other).\n\nAnd for Git's test suite, we simply do not control the name of the\ndirectory in which the Git executable lives (think: `git clone\n junios-git`).\n\nAnd even if we could control that name: it simply is not a valid\ninstallation location. For example, it is unlikely that the translation\nlive in `../share/locale/de/LC_MESSAGES/git.mo` relative to the Git\nexecutable.\n\n> - Why is it better not to bother fixing that failure? Is it\n> because this happens only in the test helper and won't hurt end\n> users?\n> \n> I do not mind this particular \"sweeping it under the rug\" if the\n> rationale is \"it only is the strange set-up of test-tool that causes\n> it, and we shouldn't burden the code shared with the actual runtime\n> to compute runtime prefix just to fix this bug\". I do think it is\n> not productive to bend backwards if it is only done in order to work\n> around unusual setup in the test helper binary and I do agree that\n> ignoring the warning is the right solution.\n> \n> I just do not like to see that our commits do not explain why we\n> chose to ignore, instead of \"fix\", the issue.\n\nBut that was clarified already to your satisfaction in the RUNTIME_PREFIX\npatches. Or do you want to reopen that case now?\n\nCiao,\nDscho\n\n> \n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > t/t0061-run-command.sh | 3 ++-\n> > 1 file changed, 2 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> > index 99a614bc7c..5a2d087bf0 100755\n> > --- a/t/t0061-run-command.sh\n> > +++ b/t/t0061-run-command.sh\n> > @@ -166,7 +166,8 @@ test_trace () {\n> > \texpect=\"$1\"\n> > \tshift\n> > \tGIT_TRACE=1 test-tool run-command \"$@\" run-command true 2>&1 >/dev/null | \\\n> > -\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' >actual &&\n> > +\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' \\\n> > +\t\t\t-e '/RUNTIME_PREFIX requested/d' >actual &&\n> > \techo \"$expect true\" >expect &&\n> > \ttest_cmp expect actual\n> > }\n> \n"},{"id":"367899","messageId":"xmqqzhrk8k1n.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1901282337550.41@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v4 18/21] t0061: fix with --with-dashes and RUNTIME_PREFIX","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-28T23:00:20Z","receivedAt":"2019-01-28T23:00:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> Two questions that would come to the readers' minds are\n>> \n>> - Why \"it fails to detect the system prefix correctly\"? Is that a\n>> bug waiting to hurt end users?\n>\n> I recall explaining that already in a different thread (one that actually\n> *was* about the RUNTIME_PREFIX feature, where you asked why we cannot test\n> for it in the test suite): to verify that we are in a valid Git\n> installation location,...\n> ... But that was clarified already to your satisfaction in the RUNTIME_PREFIX\n> patches. Or do you want to reopen that case now?\n\nThis is not about me. It is about those who read _this_ commit ---\nif the change is justified to them clearly enough, then we'd be\nhappy. I did not think it is with the proposed log message, and\nthat is where my comment came from.\n"},{"id":"367900","messageId":"nycvar.QRO.7.76.6.1901282352470.41@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqlg35elgv.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-28T23:01:06Z","receivedAt":"2019-01-28T23:01:32Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Sun, 27 Jan 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n> \n> > No, not really. Actually, not at all.\n> >\n> >> It would be, under this new arrangement of the code, most natural for\n> >> Those who want to use gnu-time to arrange it to be somehow added to\n> >> $BREW_INSTALL_PACKAGES, no?\n> >\n> > The purpose of BREW_INSTALL_PACKAGES is to list the packages necessary to\n> > build Git and run its test suite, and the only reason why this is no\n> > longer a hard-coded list of packages is that it depends on the CI platform\n> > (or more concretely, on the available macOS agents of said CI platform)\n> > which packages need to be installed to do so.\n> >\n> > The gnu-time package is not such a package, and it is unlikely to be\n> > dependent on the particular CI you want to use.\n> \n> Those who want to do perf tests in the current setup would need to\n> install gnu-time because the current setup is only Travis, whose\n> macOS agent does not have it preinstalled. Other CI platforms'\n> macOS agents may already have it, they may not want to get an error\n> by trying to install it there. I am not sure how that is different\n> from the situation for gettext etc.?\n\nThe big difference is that gettext is needed to build Git and run its test\nsuite. While gnu-time is only needed if you want to run the perf tests,\nwhich is not a part of the CI configuration we have, neither Travis nor\nAzure Pipelines.\n\nSo as long as we do not run the perf tests as part of the CI runs, that\noptional dependency should *not* be included in *CI_TYPE* specific\nsections of the code.\n\nSince the perf test reference in this comment that you keep talking about\nis so clearly intended for some human being who wants to run the scripts\nin ci/ interactively (which pretty much contradicts the \"ci\" in the name a\nbit), I would even argue that it already is too hidden in the depths of\nthe scripts to be useful. But sticking it even deeper into the\nCI_TYPE-specific sections? That would make it *even harder* to find!\n\nSo no, I think this suggestion to move that comment into exactly those\nsections (and of course, repeat it, identically, because with this here\npatch series we support *two* CI types) is something that we really want\nto let slide.\n\nCiao,\nDscho\n"},{"id":"367901","messageId":"xmqqva288jts.fsf@gitster-ct.c.googlers.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.1901282352470.41@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v4 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-28T23:05:03Z","receivedAt":"2019-01-28T23:05:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> The big difference is that gettext is needed to build Git and run its test\n> suite. While gnu-time is only needed if you want to run the perf tests,\n> which is not a part of the CI configuration we have, neither Travis nor\n> Azure Pipelines.\n>\n> So as long as we do not run the perf tests as part of the CI runs, that\n> optional dependency should *not* be included in *CI_TYPE* specific\n> sections of the code.\n\nAh, in that case, I do not think it makes much sense to even keep\nthat comment. As you said, ci/ is about running tests under CI\nplatform, and the scripts are not designed to be run manually with\ntweaks, and none of our CI integration runs the perf thing, there is\nno point to even mention it.\n"},{"id":"367938","messageId":"nycvar.QRO.7.76.6.1901291049430.41@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqva288jts.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-29T09:54:02Z","receivedAt":"2019-01-29T09:54:27Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 28 Jan 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n> \n> > The big difference is that gettext is needed to build Git and run its test\n> > suite. While gnu-time is only needed if you want to run the perf tests,\n> > which is not a part of the CI configuration we have, neither Travis nor\n> > Azure Pipelines.\n> >\n> > So as long as we do not run the perf tests as part of the CI runs, that\n> > optional dependency should *not* be included in *CI_TYPE* specific\n> > sections of the code.\n> \n> Ah, in that case, I do not think it makes much sense to even keep\n> that comment. As you said, ci/ is about running tests under CI\n> platform, and the scripts are not designed to be run manually with\n> tweaks, and none of our CI integration runs the perf thing, there is\n> no point to even mention it.\n\nWell, yes and no.\n\nYes, it does not make much sense to cater to interactive usage when we\ntarget a CI system.\n\nBut yes, it *does* make sense to let users run those ci/ scripts in case\nthey want to investigate test failures. Take, for example, a Windows user\ncontributing patches that end up failing on Ubuntu with clang. Then they\ncan install Ubuntu from the Windows Store, run those scripts in ci/ to\npainlessly get the necessary packages (without spending extra time to\nresearch which ones, and how to install them on Ubuntu in the first\nplace), and they have a pretty good head start at debugging this\ninteractively.\n\nLikewise, I take the comment about perf testing as a seedling for testing\nperformance in a controlled, repeatable and automated manner. As such, it\nwould probably make less sense to step on that seedling and crush it;\nInstead, we will want to keep the comment there because it does no evil\nand at the same time is kind of a placeholder for something beautiful to\ncome.\n\nCiao,\nDscho\n"},{"id":"367939","messageId":"nycvar.QRO.7.76.6.1901291054280.41@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"xmqqzhrk8k1n.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 18/21] t0061: fix with --with-dashes and RUNTIME_PREFIX","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-01-29T09:55:21Z","receivedAt":"2019-01-29T09:55:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 28 Jan 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n> \n> >> Two questions that would come to the readers' minds are\n> >> \n> >> - Why \"it fails to detect the system prefix correctly\"? Is that a\n> >> bug waiting to hurt end users?\n> >\n> > I recall explaining that already in a different thread (one that actually\n> > *was* about the RUNTIME_PREFIX feature, where you asked why we cannot test\n> > for it in the test suite): to verify that we are in a valid Git\n> > installation location,...\n> > ... But that was clarified already to your satisfaction in the RUNTIME_PREFIX\n> > patches. Or do you want to reopen that case now?\n> \n> This is not about me. It is about those who read _this_ commit ---\n> if the change is justified to them clearly enough, then we'd be\n> happy. I did not think it is with the proposed log message, and\n> that is where my comment came from.\n\nYou know what, I finally start to see your point.\n\nWill adjust the commit message accordingly,\nDscho\n"},{"id":"367960","messageId":"f0852de8abe2a501fdb829b1c241f02fa1deeaf0.1548771560.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 02/21] ci: rename the library of common functions","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:23Z","receivedAt":"2019-01-29T14:19:26Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe name is hard-coded to reflect that we use Travis CI for continuous\ntesting.\n\nIn the next commits, we will extend this to be able use Azure DevOps,\ntoo.\n\nSo let's adjust the name to make it more generic.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 2 +-\n ci/{lib-travisci.sh => lib.sh} | 0\n ci/print-test-failures.sh | 2 +-\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 2 +-\n 8 files changed, 7 insertions(+), 7 deletions(-)\n rename ci/{lib-travisci.sh => lib.sh} (100%)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 06c3546e1e..fe65144152 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -3,7 +3,7 @@\n # Install dependencies required to build and test Git on Linux and macOS\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\ndiff --git a/ci/lib-travisci.sh b/ci/lib.sh\nsimilarity index 100%\nrename from ci/lib-travisci.sh\nrename to ci/lib.sh\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex d55460a212..7aef39a2fd 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -3,7 +3,7 @@\n # Print output of failing tests\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n # Tracing executed commands would produce too much noise in the loop below.\n set +x\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex cda170d5c2..db342bb6a8 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -3,7 +3,7 @@\n # Build and test Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n ln -s \"$cache_dir/.prove\" t/.prove\n \ndiff --git a/ci/run-linux32-docker.sh b/ci/run-linux32-docker.sh\nindex 21637903ce..751acfcf8a 100755\n--- a/ci/run-linux32-docker.sh\n+++ b/ci/run-linux32-docker.sh\n@@ -3,7 +3,7 @@\n # Download and run Docker image to build and test 32-bit Git\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n docker pull daald/ubuntu32:xenial\n \ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\nindex 5688f261d0..dc189c7456 100755\n--- a/ci/run-static-analysis.sh\n+++ b/ci/run-static-analysis.sh\n@@ -3,7 +3,7 @@\n # Perform various static code analysis checks\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n make --jobs=2 coccicheck\n \ndiff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh\nindex d99a180e52..a73a4eca0a 100755\n--- a/ci/run-windows-build.sh\n+++ b/ci/run-windows-build.sh\n@@ -6,7 +6,7 @@\n # supported) and a commit hash.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n test $# -ne 2 && echo \"Unexpected number of parameters\" && exit 1\n test -z \"$GFW_CI_TOKEN\" && echo \"GFW_CI_TOKEN not defined\" && exit\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex a20de9ca12..d3cdbac73f 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -3,7 +3,7 @@\n # Perform sanity checks on documentation and build it.\n #\n \n-. ${0%/*}/lib-travisci.sh\n+. ${0%/*}/lib.sh\n \n gem install asciidoctor\n \n-- \ngitgitgadget\n\n"},{"id":"367961","messageId":"pull.31.v6.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v5.git.gitgitgadget@gmail.com","subject":"[PATCH v6 00/21] Offer to run CI/PR builds in Azure Pipelines","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:21Z","receivedAt":"2019-01-29T14:19:28Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"For a long time already, we have tested Git's source code continuously via\nTravis CI, see e.g. https://travis-ci.org/git/git/builds/421738884. It has\nserved us well, and more and more developers actually pay attention and\nbenefit from the testing this gives us.\n\nIt is also an invaluable tool for contributors who can validate their code\ncontributions via PRs on GitHub, e.g. to verify that their tests do actually\nrun on macOS (i.e. with the BSD family of Unix tools instead of the GNU\none).\n\nThe one sad part about this is the Windows support. Travis used to lack it\n(Travis announced some early support for Windows, hot off the press\n[https://blog.travis-ci.com/2018-10-11-windows-early-release]), and we work\naround that by using Azure Pipelines (the CI part of Azure DevOps, formerly\nknown as Visual Studio Team Services) indirectly: one phase in Travis would\ntrigger a build, wait for its log, and then paste that log.\n\nAs Git's Windows builds (and tests!) take quite a bit of time, Travis often\ntimed out, or somehow the trigger did not work, and for security reasons\n(the Windows builds are performed in a private pool of containers), the\nWindows builds are completely disabled for Pull Requests on GitHub.\n\nOne might ask why we did not use Azure Pipelines directly. There were a\ncouple of reasons for that:\n\n * most notably, Azure Pipelines' build logs could not be viewed\n anonymously,\n * while Azure Pipelines had Linux and Windows agents, it lacked macOS\n agents,\n * etc\n\nThe main two reasons no longer apply: macOS agents are available now\n[https://docs.microsoft.com/en-us/azure/devops/release-notes/2018/jul-10-vsts]\n, as are public projects\n[https://docs.microsoft.com/en-us/azure/devops/organizations/public/about-public-projects]\n, i.e. it is possible to configure a Azure Pipelines project so that anybody \ncan view the logs. Since I offered v1, Azure Pipelines has been made\navailable via the GitHub Marketplace, free of cost for open source projects.\n\nI had secured such a public project for Git for Windows already, and I also\ngot one for Git. For now, the latter is hooked up with my personal git.git\nfork on GitHub, but it is my hope that I convince y'all that these Azure\nPipelines builds are a good idea, and then hook it up with \nhttps://github.com/git/git.\n\nAs a special treat, this patch series adds the ability to present the\noutcome of Git's test suite as JUnit-style .xml files. This allows the Azure\nPipelines build to present fun diagrams, trends, and makes it a lot easier\nto drill down to test failures than before. See for example \nhttps://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details\n[https://dev.azure.com/git/git/_build/results?buildId=113&view=ms.vss-test-web.test-result-details] \n(you can click on the label of the failed test, and then see the detailed\noutput in the right pane).\n\nBut maybe you're not interested as much in the presentation of test failures\nas in the speed? Well, I got you covered with that, too. As of v3, the test\nsuite is run in parallel on Windows, cutting the overall run time to ~33\nminutes (see one of the latest builds\n[https://dev.azure.com/git/git/_build/results?buildId=311&view=logs], for\nexample).\n\nThis patch series took waaaaaaaay more time than I had originally\nanticipated, but I think that in particular the advanced display of the test\nresults and the reduction of the overall run time was worth it. Please let\nme know what you think about this.\n\nChanges since v5:\n\n * Fixed sparse warnings.\n * Reworded the commit message of \"t0061: fix for --with-dashes and RUN\n TIME_PREFIX\" to clarify why there is a warning message in the first\n place, why this cannot be fixed, and that we introduce a simple\n workaround here.\n\nChanges since v4:\n\n * Clarified in the commit message why BREW_INSTALL_PACKAGES is needed.\n * Added a clause to error out if we could not detect the CI type.\n * Introduced a CI_TYPE variable to guard CI type-specific code.\n * MAKEFLAGS is now actually configured in the CI type-specific section.\n * Moved the inadvertent fixup for GIT_TEST_OPTS from \"Add a build\n definition for Azure DevOps\" to \"ci/lib.sh: add support for Azure\n Pipelines\".\n * Backed out the Windows support from the commit adding the initial Azure\n Pipelines support, instead adding it later to help the flow of the patch\n series.\n * Clarified in \"test-date: add a subcommand to measure times in shell\n scripts\"'s commit message that one of the goals is portability.\n * Moved all of those CI type-specific definitions in ci/lib.sh after the \n set -ex statement (as suggested by Gábor a loooong time ago already).\n * Guarded the Travis-specific way to \"publish\" trash directories of failed\n tests via base64-encoded tar files in the log; For Azure Pipelines, these\n are published as build artifacts instead.\n\nChanges since v3:\n\n * Fixed the incorrect usage of cut -c that resulted in an empty trace when\n failed tests were published (except if the very first test case in a test\n script failed, then it was correct by mistake).\n * Excluded the previous test case's \"ok\" (or \"not ok\") line from the trace\n published with failed tests.\n * Renamed TRAVIS_BRANCH to CI_BRANCH already in the first commit, as we\n should not override TRAVIS_BRANCH with something that it is not.\n * Rebased onto current master to avoid merge conflicts with the\n recently-merged sg/stress-test branch (no, Junio, I really trust myself\n more than you to resolve those merge conflicts).\n\nChanges since v2:\n\n * Removed left-over debugging code that would skip a good chunk of \n t0000-init.sh.\n * Fixed the URL of the build badge.\n * Removed a trailing empty line from, and added a missing closing pointy\n bracket to, ci/mount-fileshare.sh.\n * Moved the \"travis: fix skipping tagged releases\" commit up to the\n beginning of the patch series.\n * The commit message of \"ci/lib.sh: add support for Azure Pipelines\" now\n mentions explicitly that the Homebrew packages that need to be installed\n on Travis' macOS agents are already installed on Azure Pipelines'.\n * Some commands were not guarded by || exit 1, i.e. if they would fail, the\n build would not have failed.\n * We now install gcc-8 for the linux-gcc job.\n * We no longer try to re-run failed tests with verbose log. Instead, we\n simply use the verbose log to begin with. Tests showed that it had a\n negligible impact on the overall run time.\n * The test_atexit_handler function was scratched; It would be the right\n thing to do, but is actually an independent topic (it was only\n implemented in v2 to accommodate the \"re-run with verbose log on failure\"\n trick)\n * We now use a new YAML schema (most notably, \"phase\" is now known as\n \"job\")\n * The Windows job contained PowerShell sections that were indented with 3\n spaces instead of 2.\n * The Windows job is now structured better, by separating different\n concerns into different \"tasks\" so that it is easier to see what exactly\n failed (was it the build? was it the test?)\n * The Windows job was split into a job to build Git and 10 parallel jobs to\n run the test suite with the artifacts built by the first job. This\n reduces the overall run time from ~1h20 (which was the run time by the\n Windows job) to ~35 minutes (which is the run time of the linux-gcc job).\n * The JUnit XML is now written using a test helper rather than a fragile\n and probably not even portable sed call.\n * Since we needed to determine the file size of the verbose log (to cut out\n individual test cases' log output), we now introduce a test helper to do\n that, and use it throughout the test suite (where Perl was used before).\n * It would appear that a recent change either in Cygwin or in the Azure VMs\n causes problems sporadically where the trash directories cannot be\n removed, but a subsequent rm will succeed. We now simply do that, because\n it won't harm the common case (where the first rm succeeds already) and\n because it helps the Windows job succeed pretty reliably.\n\nChanges since v1:\n\n * Removed a superfluous eval.\n * Added the commit that fixes the Travis PR builds targeting master that \n just happens to be tagged (see \n https://travis-ci.org/git/git/jobs/424276413 for an incorrectly-skipped\n build).\n * The commit messages and the cover letter now reflect the name change from\n Visual Studio Team Services to Azure DevOps (and in particular, Azure\n Pipelines for the automated builds).\n * Now we're using test_atexit (which we introduced for that purpose)\n instead of hard-coding kill_p4d and stop_git_daemon.\n * The build should now also succeed for Pull Requests (where secret\n variables are not available, for security reasons, and as a consequence\n the file share cannot be mounted).\n * The shell scripted parts now use proper && chains.\n\nJohannes Schindelin (21):\n travis: fix skipping tagged releases\n ci: rename the library of common functions\n ci/lib.sh: encapsulate Travis-specific things\n ci: inherit --jobs via MAKEFLAGS in run-build-and-tests\n ci: use a junction on Windows instead of a symlink\n test-date: add a subcommand to measure times in shell scripts\n tests: optionally write results as JUnit-style .xml\n ci/lib.sh: add support for Azure Pipelines\n Add a build definition for Azure DevOps\n ci: add a Windows job to the Azure Pipelines definition\n ci: use git-sdk-64-minimal build artifact\n mingw: be more generous when wrapping up the setitimer() emulation\n README: add a build badge (status of the Azure Pipelines build)\n tests: avoid calling Perl just to determine file sizes\n tests: include detailed trace logs with --write-junit-xml upon failure\n mingw: try to work around issues with the test cleanup\n tests: add t/helper/ to the PATH with --with-dashes\n t0061: workaround issues with --with-dashes and RUNTIME_PREFIX\n tests: optionally skip bin-wrappers/\n ci: speed up Windows phase\n ci: parallelize testing on Windows\n\n Makefile | 11 +\n README.md | 2 +\n azure-pipelines.yml | 387 ++++++++++++++++++++++++++++\n ci/install-dependencies.sh | 5 +-\n ci/{lib-travisci.sh => lib.sh} | 92 +++++--\n ci/make-test-artifacts.sh | 12 +\n ci/mount-fileshare.sh | 25 ++\n ci/print-test-failures.sh | 15 +-\n ci/run-build-and-tests.sh | 9 +-\n ci/run-linux32-build.sh | 2 +-\n ci/run-linux32-docker.sh | 2 +-\n ci/run-static-analysis.sh | 4 +-\n ci/run-test-slice.sh | 17 ++\n ci/run-windows-build.sh | 2 +-\n ci/test-documentation.sh | 7 +-\n compat/mingw.c | 2 +-\n t/.gitignore | 1 +\n t/README | 9 +\n t/helper/test-date.c | 12 +\n t/helper/test-path-utils.c | 64 +++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/helper/test-xml-encode.c | 80 ++++++\n t/t0021-conversion.sh | 2 +-\n t/t0061-run-command.sh | 3 +-\n t/t1050-large.sh | 2 +-\n t/t5315-pack-objects-compression.sh | 2 +-\n t/t9303-fast-import-compression.sh | 2 +-\n t/test-lib.sh | 138 +++++++++-\n 29 files changed, 862 insertions(+), 49 deletions(-)\n create mode 100644 azure-pipelines.yml\n rename ci/{lib-travisci.sh => lib.sh} (50%)\n create mode 100755 ci/make-test-artifacts.sh\n create mode 100755 ci/mount-fileshare.sh\n create mode 100755 ci/run-test-slice.sh\n create mode 100644 t/helper/test-xml-encode.c\n\n\nbase-commit: 16a465bc018d09e9d7bbbdc5f40a7fb99c21f8ef\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-31%2Fdscho%2Fvsts-ci-v6\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-31/dscho/vsts-ci-v6\nPull-Request: https://github.com/gitgitgadget/git/pull/31\n\nRange-diff vs v5:\n\n 1: f553fd4fb3 = 1: f553fd4fb3 travis: fix skipping tagged releases\n 2: f0852de8ab = 2: f0852de8ab ci: rename the library of common functions\n 3: 7c16d31b46 = 3: 7c16d31b46 ci/lib.sh: encapsulate Travis-specific things\n 4: bf72fb1004 = 4: bf72fb1004 ci: inherit --jobs via MAKEFLAGS in run-build-and-tests\n 5: 681f8e65b4 = 5: 681f8e65b4 ci: use a junction on Windows instead of a symlink\n 6: ccf8bf53d7 = 6: ccf8bf53d7 test-date: add a subcommand to measure times in shell scripts\n 7: 31fbe11425 ! 7: e9a869df57 tests: optionally write results as JUnit-style .xml\n @@ -91,7 +91,7 @@\n +\t\tif (tmp2) {\n +\t\t\tif ((ch & 0xc0) != 0x80) {\n +\t\t\t\tfputs(utf8_replace_character, stdout);\n -+\t\t\t\ttmp2 = 0;\n ++\t\t\t\ttmp2 = NULL;\n +\t\t\t\tcur--;\n +\t\t\t\tcontinue;\n +\t\t\t}\n @@ -99,7 +99,7 @@\n +\t\t\ttmp2++;\n +\t\t\tif (--remaining == 0) {\n +\t\t\t\tfwrite(tmp, tmp2 - tmp, 1, stdout);\n -+\t\t\t\ttmp2 = 0;\n ++\t\t\t\ttmp2 = NULL;\n +\t\t\t}\n +\t\t\tcontinue;\n +\t\t}\n 8: 47fe8aa84a = 8: 4c78085af7 ci/lib.sh: add support for Azure Pipelines\n 9: 6044b919b0 = 9: d34812bdf6 Add a build definition for Azure DevOps\n 10: 3a2993c826 = 10: d565131781 ci: add a Windows job to the Azure Pipelines definition\n 11: 52d63018c9 = 11: 1ab1d14e06 ci: use git-sdk-64-minimal build artifact\n 12: 1cef14e4c0 = 12: c1ab8df677 mingw: be more generous when wrapping up the setitimer() emulation\n 13: eda46863e8 = 13: b6316e1f7d README: add a build badge (status of the Azure Pipelines build)\n 14: 4303bbd4e8 = 14: 7a5caa2e0f tests: avoid calling Perl just to determine file sizes\n 15: dfd3516f39 = 15: 2593b9ba7e tests: include detailed trace logs with --write-junit-xml upon failure\n 16: 9984c6d17b = 16: 991b41afa4 mingw: try to work around issues with the test cleanup\n 17: a8101ae3e1 = 17: 77896b2f8e tests: add t/helper/ to the PATH with --with-dashes\n 18: 7518266a19 ! 18: 4ec6cc83ef t0061: fix with --with-dashes and RUNTIME_PREFIX\n @@ -1,23 +1,29 @@\n Author: Johannes Schindelin \n \n - t0061: fix with --with-dashes and RUNTIME_PREFIX\n + t0061: workaround issues with --with-dashes and RUNTIME_PREFIX\n \n When building Git with RUNTIME_PREFIX and starting a test helper from\n - t/helper/, it fails to detect the system prefix correctly.\n -\n - This is the reason that the warning\n + t/helper/, it fails to detect a system prefix. The reason is that the\n + RUNTIME_PREFIX feature wants to use the location of the Git executable\n + to determine where the support files can be found, e.g. system-wide Git\n + config or the translations. This does not make any sense for the test\n + helpers, though, as they are distinctly not in a directory structure\n + resembling the final installation location of Git.\n +\n + That is the reason why the test helpers rely on environment variables to\n + indicate the location of the needed support files, e.g.\n + GIT_TEXTDOMAINDIR. If this information is missing, the output will\n + contain warnings like this one:\n \n RUNTIME_PREFIX requested, but prefix computation failed. [...]\n \n - to be printed.\n -\n - In t0061, we did not expect that to happen, and it actually did not\n - happen in the normal case, because bin-wrappers/test-tool specifically\n - sets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool wants\n - to know about the runtime prefix).\n + In t0061, we did not expect that to happen, and it actually does not\n + happen in the regular case, because bin-wrappers/test-tool specifically\n + sets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool needs\n + to know anything about any runtime prefix).\n \n However, with --with-dashes, bin-wrappers/test-tool is no longer called,\n - but t/helper/test-tool is called directly.\n + but t/helper/test-tool is called directly instead.\n \n So let's just ignore the RUNTIME_PREFIX warning.\n \n 19: 6a8adbe8ba = 19: 248473d9fa tests: optionally skip bin-wrappers/\n 20: b39e16527d = 20: 3532811a49 ci: speed up Windows phase\n 21: 6ff87d1772 = 21: 1572444361 ci: parallelize testing on Windows\n\n-- \ngitgitgadget\n"},{"id":"367962","messageId":"f553fd4fb37c26e04b404b0819d3f7bcd51cd4f2.1548771560.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 01/21] travis: fix skipping tagged releases","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:22Z","receivedAt":"2019-01-29T14:19:29Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWhen building a PR, TRAVIS_BRANCH refers to the *target branch*.\nTherefore, if a PR targets `master`, and `master` happened to be tagged,\nwe skipped the build by mistake.\n\nFix this by using TRAVIS_PULL_REQUEST_BRANCH (i.e. the *source branch*)\nwhen available, falling back to TRAVIS_BRANCH (i.e. for CI builds, also\nknown as \"push builds\").\n\nLet's give it a new variable name, too: CI_BRANCH (as it is different\nfrom TRAVIS_BRANCH). This also prepares for the upcoming patches which\nwill make our ci/* code a bit more independent from Travis and open it\nto other CI systems (in particular to Azure Pipelines).\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib-travisci.sh | 25 ++++++++++++++-----------\n 1 file changed, 14 insertions(+), 11 deletions(-)\n\ndiff --git a/ci/lib-travisci.sh b/ci/lib-travisci.sh\nindex 69dff4d1ec..9c6ddeb374 100755\n--- a/ci/lib-travisci.sh\n+++ b/ci/lib-travisci.sh\n@@ -5,18 +5,17 @@ skip_branch_tip_with_tag () {\n \t# at the same commit as the tip of the branch is pushed, and building\n \t# both at the same time is a waste.\n \t#\n-\t# Travis gives a tagname e.g. v2.14.0 in $TRAVIS_BRANCH when\n-\t# the build is triggered by a push to a tag. Let's see if\n-\t# $TRAVIS_BRANCH is exactly at a tag, and if so, if it is\n-\t# different from $TRAVIS_BRANCH. That way, we can tell if\n-\t# we are building the tip of a branch that is tagged and\n-\t# we can skip the build because we won't be skipping a build\n-\t# of a tag.\n-\n-\tif TAG=$(git describe --exact-match \"$TRAVIS_BRANCH\" 2>/dev/null) &&\n-\t\ttest \"$TAG\" != \"$TRAVIS_BRANCH\"\n+\t# When the build is triggered by a push to a tag, $CI_BRANCH will\n+\t# have that tagname, e.g. v2.14.0. Let's see if $CI_BRANCH is\n+\t# exactly at a tag, and if so, if it is different from $CI_BRANCH.\n+\t# That way, we can tell if we are building the tip of a branch that\n+\t# is tagged and we can skip the build because we won't be skipping a\n+\t# build of a tag.\n+\n+\tif TAG=$(git describe --exact-match \"$CI_BRANCH\" 2>/dev/null) &&\n+\t\ttest \"$TAG\" != \"$CI_BRANCH\"\n \tthen\n-\t\techo \"$(tput setaf 2)Tip of $TRAVIS_BRANCH is exactly at $TAG$(tput sgr0)\"\n+\t\techo \"$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)\"\n \t\texit 0\n \tfi\n }\n@@ -81,6 +80,10 @@ check_unignored_build_artifacts ()\n # and installing dependencies.\n set -ex\n \n+# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n+# want here. We want the source branch instead.\n+CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+\n cache_dir=\"$HOME/travis-cache\"\n good_trees_file=\"$cache_dir/good-trees\"\n \n-- \ngitgitgadget\n\n"},{"id":"367964","messageId":"bf72fb10049283cb0d85ab3906f1ef535cb64ff8.1548771560.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 04/21] ci: inherit --jobs via MAKEFLAGS in run-build-and-tests","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:25Z","receivedAt":"2019-01-29T14:19:29Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nLet's not decide in the generic ci/ part how many jobs to run in\nparallel; different CI configurations would favor a different number of\nparallel jobs, and it is easy enough to hand that information down via\nthe `MAKEFLAGS` variable.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 1 +\n ci/run-build-and-tests.sh | 2 +-\n ci/run-linux32-build.sh | 2 +-\n ci/run-static-analysis.sh | 2 +-\n ci/test-documentation.sh | 4 ++--\n 5 files changed, 6 insertions(+), 5 deletions(-)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 3f286d86a6..32a28fd209 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -101,6 +101,7 @@ then\n \tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+\texport MAKEFLAGS=\"--jobs=2\"\n else\n \techo \"Could not identify CI type\" >&2\n \texit 1\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex db342bb6a8..80d72d120f 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -7,7 +7,7 @@\n \n ln -s \"$cache_dir/.prove\" t/.prove\n \n-make --jobs=2\n+make\n make --quiet test\n if test \"$jobname\" = \"linux-gcc\"\n then\ndiff --git a/ci/run-linux32-build.sh b/ci/run-linux32-build.sh\nindex 2c60d2e70a..09e9276e12 100755\n--- a/ci/run-linux32-build.sh\n+++ b/ci/run-linux32-build.sh\n@@ -55,6 +55,6 @@ linux32 --32bit i386 su -m -l $CI_USER -c '\n \tset -ex\n \tcd /usr/src/git\n \ttest -n \"$cache_dir\" && ln -s \"$cache_dir/.prove\" t/.prove\n-\tmake --jobs=2\n+\tmake\n \tmake --quiet test\n '\ndiff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh\nindex dc189c7456..a19aa7ebbc 100755\n--- a/ci/run-static-analysis.sh\n+++ b/ci/run-static-analysis.sh\n@@ -5,7 +5,7 @@\n \n . ${0%/*}/lib.sh\n \n-make --jobs=2 coccicheck\n+make coccicheck\n \n set +x\n \ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex 7d0beb2832..be3b7d376a 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -12,7 +12,7 @@ make check-builtins\n make check-docs\n \n # Build docs with AsciiDoc\n-make --jobs=2 doc > >(tee stdout.log) 2> >(tee stderr.log >&2)\n+make doc > >(tee stdout.log) 2> >(tee stderr.log >&2)\n ! test -s stderr.log\n test -s Documentation/git.html\n test -s Documentation/git.xml\n@@ -24,7 +24,7 @@ check_unignored_build_artifacts\n \n # Build docs with AsciiDoctor\n make clean\n-make --jobs=2 USE_ASCIIDOCTOR=1 doc > >(tee stdout.log) 2> >(tee stderr.log >&2)\n+make USE_ASCIIDOCTOR=1 doc > >(tee stdout.log) 2> >(tee stderr.log >&2)\n sed '/^GIT_VERSION = / d' stderr.log\n ! test -s stderr.log\n test -s Documentation/git.html\n-- \ngitgitgadget\n\n"},{"id":"367963","messageId":"7c16d31b46258ad00768c6125d0c6ad64d341c15.1548771560.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 03/21] ci/lib.sh: encapsulate Travis-specific things","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:24Z","receivedAt":"2019-01-29T14:19:30Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe upcoming patches will allow building git.git via Azure Pipelines\n(i.e. Azure DevOps' Continuous Integration), where variable names and\nURLs look a bit different than in Travis CI.\n\nAlso, the configurations of the available agents are different. For\nexample, Travis' and Azure Pipelines' macOS agents are set up\ndifferently, so that on Travis, we have to install the git-lfs and\ngettext Homebrew packages, and on Azure Pipelines we do not need to.\nLikewise, Azure Pipelines' Ubuntu agents already have asciidoctor\ninstalled.\n\nFinally, on Azure Pipelines the natural way is not to base64-encode tar\nfiles of the trash directories of failed tests, but to publish build\nartifacts instead. Therefore, that code to log those base64-encoded tar\nfiles is guarded to be Travis-specific.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/install-dependencies.sh | 3 ++-\n ci/lib.sh | 45 +++++++++++++++++++++++++++-----------\n ci/print-test-failures.sh | 8 +++++++\n ci/test-documentation.sh | 1 +\n 4 files changed, 43 insertions(+), 14 deletions(-)\n\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex fe65144152..bcdcc71592 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -37,7 +37,8 @@ osx-clang|osx-gcc)\n \tbrew update --quiet\n \t# Uncomment this if you want to run perf tests:\n \t# brew install gnu-time\n-\tbrew install git-lfs gettext\n+\ttest -z \"$BREW_INSTALL_PACKAGES\" ||\n+\tbrew install $BREW_INSTALL_PACKAGES\n \tbrew link --force gettext\n \tbrew install caskroom/cask/perforce\n \t;;\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 9c6ddeb374..3f286d86a6 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -24,7 +24,7 @@ skip_branch_tip_with_tag () {\n # job if we encounter the same tree again and can provide a useful info\n # message.\n save_good_tree () {\n-\techo \"$(git rev-parse $TRAVIS_COMMIT^{tree}) $TRAVIS_COMMIT $TRAVIS_JOB_NUMBER $TRAVIS_JOB_ID\" >>\"$good_trees_file\"\n+\techo \"$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID\" >>\"$good_trees_file\"\n \t# limit the file size\n \ttail -1000 \"$good_trees_file\" >\"$good_trees_file\".tmp\n \tmv \"$good_trees_file\".tmp \"$good_trees_file\"\n@@ -34,7 +34,7 @@ save_good_tree () {\n # successfully before (e.g. because the branch got rebased, changing only\n # the commit messages).\n skip_good_tree () {\n-\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $TRAVIS_COMMIT^{tree}) \" \"$good_trees_file\")\"\n+\tif ! good_tree_info=\"$(grep \"^$(git rev-parse $CI_COMMIT^{tree}) \" \"$good_trees_file\")\"\n \tthen\n \t\t# Haven't seen this tree yet, or no cached good trees file yet.\n \t\t# Continue the build job.\n@@ -44,18 +44,18 @@ skip_good_tree () {\n \techo \"$good_tree_info\" | {\n \t\tread tree prev_good_commit prev_good_job_number prev_good_job_id\n \n-\t\tif test \"$TRAVIS_JOB_ID\" = \"$prev_good_job_id\"\n+\t\tif test \"$CI_JOB_ID\" = \"$prev_good_job_id\"\n \t\tthen\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit has already been built and tested successfully by this build job.\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\telse\n \t\t\tcat <<-EOF\n-\t\t\t$(tput setaf 2)Skipping build job for commit $TRAVIS_COMMIT.$(tput sgr0)\n+\t\t\t$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)\n \t\t\tThis commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.\n-\t\t\tThe log of that build job is available at https://travis-ci.org/$TRAVIS_REPO_SLUG/jobs/$prev_good_job_id\n+\t\t\tThe log of that build job is available at $(url_for_job_id $prev_good_job_id)\n \t\t\tTo force a re-build delete the branch's cache and then hit 'Restart job'.\n \t\t\tEOF\n \t\tfi\n@@ -80,11 +80,32 @@ check_unignored_build_artifacts ()\n # and installing dependencies.\n set -ex\n \n-# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not what we\n-# want here. We want the source branch instead.\n-CI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+if test true = \"$TRAVIS\"\n+then\n+\tCI_TYPE=travis\n+\t# When building a PR, TRAVIS_BRANCH refers to the *target* branch. Not\n+\t# what we want here. We want the source branch instead.\n+\tCI_BRANCH=\"${TRAVIS_PULL_REQUEST_BRANCH:-$TRAVIS_BRANCH}\"\n+\tCI_COMMIT=\"$TRAVIS_COMMIT\"\n+\tCI_JOB_ID=\"$TRAVIS_JOB_ID\"\n+\tCI_JOB_NUMBER=\"$TRAVIS_JOB_NUMBER\"\n+\tCI_OS_NAME=\"$TRAVIS_OS_NAME\"\n+\tCI_REPO_SLUG=\"$TRAVIS_REPO_SLUG\"\n+\n+\tcache_dir=\"$HOME/travis-cache\"\n+\n+\turl_for_job_id () {\n+\t\techo \"https://travis-ci.org/$CI_REPO_SLUG/jobs/$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\"git-lfs gettext\"\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n+else\n+\techo \"Could not identify CI type\" >&2\n+\texit 1\n+fi\n \n-cache_dir=\"$HOME/travis-cache\"\n good_trees_file=\"$cache_dir/good-trees\"\n \n mkdir -p \"$cache_dir\"\n@@ -94,13 +115,11 @@ skip_good_tree\n \n if test -z \"$jobname\"\n then\n-\tjobname=\"$TRAVIS_OS_NAME-$CC\"\n+\tjobname=\"$CI_OS_NAME-$CC\"\n fi\n \n export DEVELOPER=1\n export DEFAULT_TEST_TARGET=prove\n-export GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n-export GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n export GIT_TEST_CLONE_2GB=YesPlease\n if [ \"$jobname\" = linux-gcc ]; then\n \texport CC=gcc-8\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex 7aef39a2fd..cf321b474d 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -38,6 +38,14 @@ do\n \t\ttest_name=\"${TEST_EXIT%.exit}\"\n \t\ttest_name=\"${test_name##*/}\"\n \t\ttrash_dir=\"trash directory.$test_name\"\n+\t\tcase \"$CI_TYPE\" in\n+\t\ttravis)\n+\t\t\t;;\n+\t\t*)\n+\t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n+\t\t\texit 1\n+\t\t\t;;\n+\t\tesac\n \t\ttrash_tgz_b64=\"trash.$test_name.base64\"\n \t\tif [ -d \"$trash_dir\" ]\n \t\tthen\ndiff --git a/ci/test-documentation.sh b/ci/test-documentation.sh\nindex d3cdbac73f..7d0beb2832 100755\n--- a/ci/test-documentation.sh\n+++ b/ci/test-documentation.sh\n@@ -5,6 +5,7 @@\n \n . ${0%/*}/lib.sh\n \n+test -n \"$ALREADY_HAVE_ASCIIDOCTOR\" ||\n gem install asciidoctor\n \n make check-builtins\n-- \ngitgitgadget\n\n"},{"id":"367965","messageId":"ccf8bf53d7f168e3edea5e4a0cc5e15486abba22.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 06/21] test-date: add a subcommand to measure times in shell scripts","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:27Z","receivedAt":"2019-01-29T14:19:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIn the next commit, we want to teach Git's test suite to optionally\noutput test results in JUnit-style .xml files. These files contain\ninformation about the time spent. So we need a way to measure time.\n\nWhile we could use `date +%s` for that, this will give us only seconds,\ni.e. very coarse-grained timings.\n\nGNU `date` supports `date +%s.%N` (i.e. nanosecond-precision output),\nbut there is no equivalent in BSD `date` (read: on macOS, we would not\nbe able to obtain precise timings).\n\nSo let's introduce `test-tool date getnanos`, with an optional start\ntime, that outputs preciser values. Note that this might not actually\ngive us nanosecond precision on some platforms, but it will give us as\nprecise information as possible, without the portability issues of shell\ncommands.\n\nGranted, it is a bit pointless to try measuring times accurately in\nshell scripts, certainly to nanosecond precision. But it is better than\nsecond-granularity.\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-date.c | 12 ++++++++++++\n 1 file changed, 12 insertions(+)\n\ndiff --git a/t/helper/test-date.c b/t/helper/test-date.c\nindex a0837371ab..792a805374 100644\n--- a/t/helper/test-date.c\n+++ b/t/helper/test-date.c\n@@ -7,6 +7,7 @@ static const char *usage_msg = \"\\n\"\n \" test-tool date parse [date]...\\n\"\n \" test-tool date approxidate [date]...\\n\"\n \" test-tool date timestamp [date]...\\n\"\n+\" test-tool date getnanos [start-nanos]\\n\"\n \" test-tool date is64bit\\n\"\n \" test-tool date time_t-is64bit\\n\";\n \n@@ -82,6 +83,15 @@ static void parse_approx_timestamp(const char **argv, struct timeval *now)\n \t}\n }\n \n+static void getnanos(const char **argv, struct timeval *now)\n+{\n+\tdouble seconds = getnanotime() / 1.0e9;\n+\n+\tif (*argv)\n+\t\tseconds -= strtod(*argv, NULL);\n+\tprintf(\"%lf\\n\", seconds);\n+}\n+\n int cmd__date(int argc, const char **argv)\n {\n \tstruct timeval now;\n@@ -108,6 +118,8 @@ int cmd__date(int argc, const char **argv)\n \t\tparse_approxidate(argv+1, &now);\n \telse if (!strcmp(*argv, \"timestamp\"))\n \t\tparse_approx_timestamp(argv+1, &now);\n+\telse if (!strcmp(*argv, \"getnanos\"))\n+\t\tgetnanos(argv+1, &now);\n \telse if (!strcmp(*argv, \"is64bit\"))\n \t\treturn sizeof(timestamp_t) == 8 ? 0 : 1;\n \telse if (!strcmp(*argv, \"time_t-is64bit\"))\n-- \ngitgitgadget\n\n"},{"id":"367966","messageId":"d34812bdf60fd9fb0f6c1c7e2ffd5685f643bd80.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 09/21] Add a build definition for Azure DevOps","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:29Z","receivedAt":"2019-01-29T14:19:34Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis commit adds an azure-pipelines.yml file which is Azure DevOps'\nequivalent to Travis CI's .travis.yml.\n\nThe main idea is to replicate the Travis configuration as faithfully as\npossible, to make it easy to compare the Azure Pipeline builds to the\nTravis ones (spoiler: some parts, especially the macOS jobs, are way\nfaster in Azure Pileines). Meaning: the number and the order of the jobs\nadded in this commit faithfully replicates what we have in .travis.yml.\n\nNote: Our .travis.yml configuration has a Windows part that is *not*\nreplicated in the Azure Pipelines definition. The reason is easy to see:\nAs Travis cannot support our Windws needs (even with the preliminary\nWindows support that was recently added to Travis after waiting for\n*years* for that feature, our test suite would simply hit Travis'\ntimeout every single time).\n\nTo make things a bit easier to understand, we refrain from using the\n`matrix` feature here because (while it is powerful) it can be a bit\nconfusing to users who are not familiar with CI setups. Therefore, we\nuse a separate phase even for similar configurations (such as GCC vs\nClang on Linux, GCC vs Clang on macOS).\n\nAlso, we make use of the shiny new feature we just introduced where the\ntest suite can output JUnit-style .xml files. This information is made\navailable in a nice UI that allows the viewer to filter by phase and/or\ntest number, and to see trends such as: number of (failing) tests, time\nspent running the test suite, etc. (While this seemingly contradicts the\nintention to replicate the Travis configuration as faithfully as\npossible, it is just too nice to show off that capability here already.)\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 259 ++++++++++++++++++++++++++++++++++++++++++\n ci/mount-fileshare.sh | 25 ++++\n 2 files changed, 284 insertions(+)\n create mode 100644 azure-pipelines.yml\n create mode 100755 ci/mount-fileshare.sh\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nnew file mode 100644\nindex 0000000000..8cdef105c6\n--- /dev/null\n+++ b/azure-pipelines.yml\n@@ -0,0 +1,259 @@\n+resources:\n+- repo: self\n+ fetchDepth: 1\n+\n+jobs:\n+- job: linux_clang\n+ displayName: linux-clang\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2-bin &&\n+\n+ export CC=clang || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-clang'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: linux_gcc\n+ displayName: linux-gcc\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo add-apt-repository ppa:ubuntu-toolchain-r/test &&\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2 language-pack-is git-svn gcc-8 || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux-gcc'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: osx_clang\n+ displayName: osx-clang\n+ condition: succeeded()\n+ pool: Hosted macOS\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ export CC=clang\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-clang'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: osx_gcc\n+ displayName: osx-gcc\n+ condition: succeeded()\n+ pool: Hosted macOS\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ ci/install-dependencies.sh || exit 1\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'osx-gcc'\n+ platform: macOS\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: gettext_poison\n+ displayName: GETTEXT_POISON\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev &&\n+\n+ export jobname=GETTEXT_POISON || exit 1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-build-and-tests.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'gettext-poison'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: linux32\n+ displayName: Linux32\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ res=0\n+ sudo AGENT_OS=\"$AGENT_OS\" BUILD_BUILDNUMBER=\"$BUILD_BUILDNUMBER\" BUILD_REPOSITORY_URI=\"$BUILD_REPOSITORY_URI\" BUILD_SOURCEBRANCH=\"$BUILD_SOURCEBRANCH\" BUILD_SOURCEVERSION=\"$BUILD_SOURCEVERSION\" SYSTEM_PHASENAME=\"$SYSTEM_PHASENAME\" SYSTEM_TASKDEFINITIONSURI=\"$SYSTEM_TASKDEFINITIONSURI\" SYSTEM_TEAMPROJECT=\"$SYSTEM_TEAMPROJECT\" CC=$CC MAKEFLAGS=\"$MAKEFLAGS\" bash -lxc ci/run-linux32-docker.sh || res=1\n+\n+ sudo chmod a+r t/out/TEST-*.xml\n+ test ! -d t/failed-test-artifacts || sudo chmod a+r t/failed-test-artifacts\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || res=1\n+ exit $res\n+ displayName: 'ci/run-linux32-docker.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'linux32'\n+ platform: Linux\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n+- job: static_analysis\n+ displayName: StaticAnalysis\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get install -y coccinelle &&\n+\n+ export jobname=StaticAnalysis &&\n+\n+ ci/run-static-analysis.sh || exit 1\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/run-static-analysis.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+\n+- job: documentation\n+ displayName: Documentation\n+ condition: succeeded()\n+ pool: Hosted Ubuntu 1604\n+ steps:\n+ - bash: |\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+ sudo apt-get update &&\n+ sudo apt-get install -y asciidoc xmlto asciidoctor &&\n+\n+ export ALREADY_HAVE_ASCIIDOCTOR=yes. &&\n+ export jobname=Documentation &&\n+\n+ ci/test-documentation.sh || exit 1\n+\n+ test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+ displayName: 'ci/test-documentation.sh'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\ndiff --git a/ci/mount-fileshare.sh b/ci/mount-fileshare.sh\nnew file mode 100755\nindex 0000000000..26b58a8096\n--- /dev/null\n+++ b/ci/mount-fileshare.sh\n@@ -0,0 +1,25 @@\n+#!/bin/sh\n+\n+die () {\n+\techo \"$*\" >&2\n+\texit 1\n+}\n+\n+test $# = 4 ||\n+die \"Usage: $0 \"\n+\n+mkdir -p \"$4\" || die \"Could not create $4\"\n+\n+case \"$(uname -s)\" in\n+Linux)\n+\tsudo mount -t cifs -o vers=3.0,username=\"$2\",password=\"$3\",dir_mode=0777,file_mode=0777,serverino \"$1\" \"$4\"\n+\t;;\n+Darwin)\n+\tpass=\"$(echo \"$3\" | sed -e 's/\\//%2F/g' -e 's/+/%2B/g')\" &&\n+\tmount -t smbfs,soft \"smb://$2:$pass@${1#//}\" \"$4\"\n+\t;;\n+*)\n+\tdie \"No support for $(uname -s)\"\n+\t;;\n+esac ||\n+die \"Could not mount $4\"\n-- \ngitgitgadget\n\n"},{"id":"367968","messageId":"c1ab8df6772045cbe63b2355904038a8556670c7.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 12/21] mingw: be more generous when wrapping up the setitimer() emulation","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:31Z","receivedAt":"2019-01-29T14:19:36Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nEvery once in a while, the Azure Pipeline fails with some semi-random\n\n\terror: timer thread did not terminate timely\n\nThis error message means that the thread that is used to emulate the\nsetitimer() function did not terminate within 1,000 milliseconds.\n\nThe most likely explanation (and therefore the one we should assume to\nbe true, according to Occam's Razor) is that the timeout of one second\nis simply not enough because we try to run so many tasks in parallel.\n\nSo let's give it ten seconds instead of only one. That should be enough.\n\nSigned-off-by: Johannes Schindelin \n---\n compat/mingw.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/compat/mingw.c b/compat/mingw.c\nindex b459e1a291..e0dfe8844d 100644\n--- a/compat/mingw.c\n+++ b/compat/mingw.c\n@@ -2100,7 +2100,7 @@ static void stop_timer_thread(void)\n \tif (timer_event)\n \t\tSetEvent(timer_event);\t/* tell thread to terminate */\n \tif (timer_thread) {\n-\t\tint rc = WaitForSingleObject(timer_thread, 1000);\n+\t\tint rc = WaitForSingleObject(timer_thread, 10000);\n \t\tif (rc == WAIT_TIMEOUT)\n \t\t\terror(\"timer thread did not terminate timely\");\n \t\telse if (rc != WAIT_OBJECT_0)\n-- \ngitgitgadget\n\n"},{"id":"367967","messageId":"7a5caa2e0f9b38edb6bdcb24da8fe19c2ef373f4.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 14/21] tests: avoid calling Perl just to determine file sizes","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:33Z","receivedAt":"2019-01-29T14:19:37Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIt is a bit ridiculous to spin up a full-blown Perl instance (especially\non Windows, where that means spinning up a full POSIX emulation layer,\nAKA the MSYS2 runtime) just to tell how large a given file is.\n\nSo let's just use the test-tool to do that job instead.\n\nThis command will also be used over the next commits, to allow for\ncutting out individual test cases' verbose log from the file generated\nvia --verbose-log.\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-path-utils.c | 12 ++++++++++++\n t/t0021-conversion.sh | 2 +-\n t/t1050-large.sh | 2 +-\n t/t5315-pack-objects-compression.sh | 2 +-\n t/t9303-fast-import-compression.sh | 2 +-\n 5 files changed, 16 insertions(+), 4 deletions(-)\n\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex ae091d9b3e..30211d6d64 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -291,6 +291,18 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn !!res;\n \t}\n \n+\tif (argc > 2 && !strcmp(argv[1], \"file-size\")) {\n+\t\tint res = 0, i;\n+\t\tstruct stat st;\n+\n+\t\tfor (i = 2; i < argc; i++)\n+\t\t\tif (stat(argv[i], &st))\n+\t\t\t\tres = error_errno(\"Cannot stat '%s'\", argv[i]);\n+\t\t\telse\n+\t\t\t\tprintf(\"%\"PRIuMAX\"\\n\", (uintmax_t)st.st_size);\n+\t\treturn !!res;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex fd5f1ac649..e10f5f787f 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -24,7 +24,7 @@ generate_random_characters () {\n }\n \n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n filter_git () {\ndiff --git a/t/t1050-large.sh b/t/t1050-large.sh\nindex 1a9b21b293..dcb4dbba67 100755\n--- a/t/t1050-large.sh\n+++ b/t/t1050-large.sh\n@@ -8,7 +8,7 @@ test_description='adding and checking out large blobs'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n test_expect_success setup '\ndiff --git a/t/t5315-pack-objects-compression.sh b/t/t5315-pack-objects-compression.sh\nindex 34c47dae09..df970d7584 100755\n--- a/t/t5315-pack-objects-compression.sh\n+++ b/t/t5315-pack-objects-compression.sh\n@@ -7,7 +7,7 @@ test_description='pack-object compression configuration'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n test_expect_success setup '\ndiff --git a/t/t9303-fast-import-compression.sh b/t/t9303-fast-import-compression.sh\nindex 856219f46a..5045f02a53 100755\n--- a/t/t9303-fast-import-compression.sh\n+++ b/t/t9303-fast-import-compression.sh\n@@ -6,7 +6,7 @@ test_description='compression setting of fast-import utility'\n # This should be moved to test-lib.sh together with the\n # copy in t0021 after both topics have graduated to 'master'.\n file_size () {\n-\tperl -e 'print -s $ARGV[0]' \"$1\"\n+\ttest-tool path-utils file-size \"$1\"\n }\n \n import_large () {\n-- \ngitgitgadget\n\n"},{"id":"367969","messageId":"2593b9ba7e396cdf84263ce6bbf192b7e91646f3.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 15/21] tests: include detailed trace logs with --write-junit-xml upon failure","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:34Z","receivedAt":"2019-01-29T14:19:38Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe JUnit XML format lends itself to be presented in a powerful UI,\nwhere you can drill down to the information you are interested in very\nquickly.\n\nFor test failures, this usually means that you want to see the detailed\ntrace of the failing tests.\n\nWith Travis CI, we passed the `--verbose-log` option to get those\ntraces. However, that seems excessive, as we do not need/use the logs in\nalmost all of those cases: only when a test fails do we have a way to\ninclude the trace.\n\nSo let's do something different when using Azure DevOps: let's run all\nthe tests with `--quiet` first, and only if a failure is encountered,\ntry to trace the commands as they are executed.\n\nOf course, we cannot turn on `--verbose-log` after the fact. So let's\njust re-run the test with all the same options, adding `--verbose-log`.\nAnd then munging the output file into the JUnit XML on the fly.\n\nNote: there is an off chance that re-running the test in verbose mode\n\"fixes\" the failures (and this does happen from time to time!). That is\na possibility we should be able to live with. Ideally, we would label\nthis as \"Passed upon rerun\", and Azure Pipelines even know about that\noutcome, but it is not available when using the JUnit XML format for\nnow:\nhttps://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs\n\nSigned-off-by: Johannes Schindelin \n---\n t/helper/test-path-utils.c | 21 +++++++++++++++++++++\n t/test-lib.sh | 22 +++++++++++++++++++++-\n 2 files changed, 42 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex 30211d6d64..6efde6f5ba 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -303,6 +303,27 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn !!res;\n \t}\n \n+\tif (argc == 4 && !strcmp(argv[1], \"skip-n-bytes\")) {\n+\t\tint fd = open(argv[2], O_RDONLY), offset = atoi(argv[3]);\n+\t\tchar buffer[65536];\n+\n+\t\tif (fd < 0)\n+\t\t\tdie_errno(\"could not open '%s'\", argv[2]);\n+\t\tif (lseek(fd, offset, SEEK_SET) < 0)\n+\t\t\tdie_errno(\"could not skip %d bytes\", offset);\n+\t\tfor (;;) {\n+\t\t\tssize_t count = read(fd, buffer, sizeof(buffer));\n+\t\t\tif (count < 0)\n+\t\t\t\tdie_errno(\"could not read '%s'\", argv[2]);\n+\t\t\tif (!count)\n+\t\t\t\tbreak;\n+\t\t\tif (write(1, buffer, count) < 0)\n+\t\t\t\tdie_errno(\"could not write to stdout\");\n+\t\t}\n+\t\tclose(fd);\n+\t\treturn 0;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a3b2166cb5..f31a1c8f79 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -639,8 +639,19 @@ test_failure_ () {\n \t\tjunit_insert=\"\"\n \t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n-\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\t\t\"$(if test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\t\t then\n+\t\t\t\ttest-tool path-utils skip-n-bytes \\\n+\t\t\t\t\t\"$GIT_TEST_TEE_OUTPUT_FILE\" $GIT_TEST_TEE_OFFSET\n+\t\t\t else\n+\t\t\t\tprintf '%s\\n' \"$@\" | sed 1d\n+\t\t\t fi)\")\"\n \t\tjunit_insert=\"$junit_insert\"\n+\t\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\t\tthen\n+\t\t\tjunit_insert=\"$junit_insert$(xml_attr_encode \\\n+\t\t\t\t\"$(cat \"$GIT_TEST_TEE_OUTPUT_FILE\")\")\"\n+\t\tfi\n \t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n \tfi\n \ttest_failure=$(($test_failure + 1))\n@@ -931,6 +942,11 @@ test_finish_ () {\n \techo >&3 \"\"\n \tmaybe_teardown_valgrind\n \tmaybe_teardown_verbose\n+\tif test -n \"$GIT_TEST_TEE_OFFSET\"\n+\tthen\n+\t\tGIT_TEST_TEE_OFFSET=$(test-tool path-utils file-size \\\n+\t\t\t\"$GIT_TEST_TEE_OUTPUT_FILE\")\n+\tfi\n }\n \n test_skip () {\n@@ -1280,6 +1296,10 @@ then\n \t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n \twrite_junit_xml --truncate \"\" \" \"\n \tjunit_suite_start=$(test-tool date getnanos)\n+\tif test -n \"$GIT_TEST_TEE_OUTPUT_FILE\"\n+\tthen\n+\t\tGIT_TEST_TEE_OFFSET=0\n+\tfi\n fi\n \n # Provide an implementation of the 'yes' utility\n-- \ngitgitgadget\n\n"},{"id":"367970","messageId":"991b41afa4a83a73f59a72504d269d64d12ecf8f.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 16/21] mingw: try to work around issues with the test cleanup","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:35Z","receivedAt":"2019-01-29T14:19:40Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nIt seems that every once in a while in the Git for Windows SDK, there\nare some transient file locking issues preventing the test clean up to\ndelete the trash directory. Let's be gentle and try again five seconds\nlater, and only error out if it still fails the second time.\n\nThis change helps Windows, and does not hurt any other platform\n(normally, it is highly unlikely that said deletion fails, and if it\ndoes, normally it will fail again even 5 seconds later).\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 6 +++++-\n 1 file changed, 5 insertions(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex f31a1c8f79..9c0ca5effb 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1104,7 +1104,11 @@ test_done () {\n \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n \n \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n-\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n+\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n+\t\t\t\t# try again in a bit\n+\t\t\t\tsleep 5;\n+\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n+\t\t\t} ||\n \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n \t\tfi\n \t\ttest_at_end_hook_\n-- \ngitgitgadget\n\n"},{"id":"367971","messageId":"3532811a49e6fc6beb7aba84b851ba4999cb0be6.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 20/21] ci: speed up Windows phase","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:38Z","receivedAt":"2019-01-29T14:19:42Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nAs Unix shell scripting comes at a hefty price on Windows, we have to\nsee where we can save some time to run the test suite.\n\nLet's skip the chain linting and the bin-wrappers/ redirection on\nWindows; this seems to shave of anywhere between 10-30% from the overall\nruntime.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 5505776876..c2bc6c68b9 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -127,6 +127,8 @@ then\n \texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n \texport MAKEFLAGS=\"--jobs=10\"\n+\ttest windows_nt != \"$CI_OS_NAME\" ||\n+\tGIT_TEST_OPTS=\"--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS\"\n else\n \techo \"Could not identify CI type\" >&2\n \texit 1\n-- \ngitgitgadget\n\n"},{"id":"367972","messageId":"77896b2f8e6fcef5fbcd9805b724f3f074c2d769.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 17/21] tests: add t/helper/ to the PATH with --with-dashes","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:35Z","receivedAt":"2019-01-29T14:19:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWe really need to be able to find the test helpers... Really. This\nchange was forgotten when we moved the test helpers into t/helper/\n\nSigned-off-by: Johannes Schindelin \n---\n t/test-lib.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9c0ca5effb..c790e98fd2 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1227,7 +1227,7 @@ else # normal case, use ../bin-wrappers only unless $with_dashes:\n \tGIT_EXEC_PATH=$GIT_BUILD_DIR\n \tif test -n \"$with_dashes\"\n \tthen\n-\t\tPATH=\"$GIT_BUILD_DIR:$PATH\"\n+\t\tPATH=\"$GIT_BUILD_DIR:$GIT_BUILD_DIR/t/helper:$PATH\"\n \tfi\n fi\n GIT_TEMPLATE_DIR=\"$GIT_BUILD_DIR\"/templates/blt\n-- \ngitgitgadget\n\n"},{"id":"367973","messageId":"248473d9fa98333d2bd122a470c2a17da859c686.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 19/21] tests: optionally skip bin-wrappers/","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:37Z","receivedAt":"2019-01-29T14:19:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis speeds up the tests by a bit on Windows, where running Unix shell\nscripts (and spawning processes) is not exactly a cheap operation.\n\nSigned-off-by: Johannes Schindelin \n---\n t/README | 9 +++++++++\n t/test-lib.sh | 19 +++++++++++++------\n 2 files changed, 22 insertions(+), 6 deletions(-)\n\ndiff --git a/t/README b/t/README\nindex 11ce7675e3..063530234f 100644\n--- a/t/README\n+++ b/t/README\n@@ -170,6 +170,15 @@ appropriately before running \"make\".\n \timplied by other options like --valgrind and\n \tGIT_TEST_INSTALLED.\n \n+--no-bin-wrappers::\n+\tBy default, the test suite uses the wrappers in\n+\t`../bin-wrappers/` to execute `git` and friends. With this option,\n+\t`../git` and friends are run directly. This is not recommended\n+\tin general, as the wrappers contain safeguards to ensure that no\n+\tfiles from an installed Git are used, but can speed up test runs\n+\tespecially on platforms where running shell scripts is expensive\n+\t(most notably, Windows).\n+\n --root=::\n \tCreate \"trash\" directories used to store all temporary data during\n \ttesting under , instead of the t/ directory.\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex c790e98fd2..25e649c997 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -111,6 +111,8 @@ do\n \t\ttest -z \"$HARNESS_ACTIVE\" && quiet=t ;;\n \t--with-dashes)\n \t\twith_dashes=t ;;\n+\t--no-bin-wrappers)\n+\t\tno_bin_wrappers=t ;;\n \t--no-color)\n \t\tcolor= ;;\n \t--va|--val|--valg|--valgr|--valgri|--valgrin|--valgrind)\n@@ -1214,16 +1216,21 @@ then\n \tPATH=$GIT_TEST_INSTALLED:$GIT_BUILD_DIR/t/helper:$PATH\n \tGIT_EXEC_PATH=${GIT_TEST_EXEC_PATH:-$GIT_EXEC_PATH}\n else # normal case, use ../bin-wrappers only unless $with_dashes:\n-\tgit_bin_dir=\"$GIT_BUILD_DIR/bin-wrappers\"\n-\tif ! test -x \"$git_bin_dir/git\"\n+\tif test -n \"$no_bin_wrappers\"\n \tthen\n-\t\tif test -z \"$with_dashes\"\n+\t\twith_dashes=t\n+\telse\n+\t\tgit_bin_dir=\"$GIT_BUILD_DIR/bin-wrappers\"\n+\t\tif ! test -x \"$git_bin_dir/git\"\n \t\tthen\n-\t\t\tsay \"$git_bin_dir/git is not executable; using GIT_EXEC_PATH\"\n+\t\t\tif test -z \"$with_dashes\"\n+\t\t\tthen\n+\t\t\t\tsay \"$git_bin_dir/git is not executable; using GIT_EXEC_PATH\"\n+\t\t\tfi\n+\t\t\twith_dashes=t\n \t\tfi\n-\t\twith_dashes=t\n+\t\tPATH=\"$git_bin_dir:$PATH\"\n \tfi\n-\tPATH=\"$git_bin_dir:$PATH\"\n \tGIT_EXEC_PATH=$GIT_BUILD_DIR\n \tif test -n \"$with_dashes\"\n \tthen\n-- \ngitgitgadget\n\n"},{"id":"367974","messageId":"1572444361982199fdab9c6f6b7e94383717b6c9.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 21/21] ci: parallelize testing on Windows","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:38Z","receivedAt":"2019-01-29T14:19:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThe fact that Git's test suite is implemented in Unix shell script that\nis as portable as we can muster, combined with the fact that Unix shell\nscripting is foreign to Windows (and therefore has to be emulated),\nresults in pretty abysmal speed of the test suite on that platform, for\npretty much no other reason than that language choice.\n\nFor comparison: while the Linux build & test is typically done within\nabout 8 minutes, the Windows build & test typically lasts about 80\nminutes in Azure Pipelines.\n\nTo help with that, let's use the Azure Pipeline feature where you can\nparallelize jobs, make jobs depend on each other, and pass artifacts\nbetween them.\n\nThe tests are distributed using the following heuristic: listing all\ntest scripts ordered by size in descending order (as a cheap way to\nestimate the overall run time), every Nth script is run (where N is the\ntotal number of parallel jobs), starting at the index corresponding to\nthe parallel job. This slicing is performed by a new function that is\nadded to the `test-tool`.\n\nTo optimize the overall runtime of the entire Pipeline, we need to move\nthe Windows jobs to the beginning (otherwise there would be a very\ndecent chance for the Pipeline to be run only the Windows build, while\nall the parallel Windows test jobs wait for this single one).\n\nWe use Azure Pipelines Artifacts for both the minimal Git for Windows\nSDK as well as the built executables, as deduplication and caching close\nto the agents makes that really fast. For comparison: while downloading\nand unpacking the minimal Git for Windows SDK via PowerShell takes only\none minute (down from anywhere between 2.5 to 7 when using a shallow\nclone), uploading it as Pipeline Artifact takes less than 30s and\ndownloading and unpacking less than 20s (sometimes even as little as\nonly twelve seconds).\n\nSigned-off-by: Johannes Schindelin \n---\n Makefile | 10 +++++\n azure-pipelines.yml | 79 ++++++++++++++++++++++++++++++++++----\n ci/make-test-artifacts.sh | 12 ++++++\n ci/run-test-slice.sh | 17 ++++++++\n t/helper/test-path-utils.c | 31 +++++++++++++++\n 5 files changed, 141 insertions(+), 8 deletions(-)\n create mode 100755 ci/make-test-artifacts.sh\n create mode 100755 ci/run-test-slice.sh\n\ndiff --git a/Makefile b/Makefile\nindex 044b4f77bd..daa318fe17 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -2927,6 +2927,16 @@ rpm::\n \t@false\n .PHONY: rpm\n \n+artifacts-tar:: $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) $(OTHER_PROGRAMS) \\\n+\t\tGIT-BUILD-OPTIONS $(TEST_PROGRAMS) $(test_bindir_programs) \\\n+\t\t$(NO_INSTALL) $(MOFILES)\n+\t$(QUIET_SUBDIR0)templates $(QUIET_SUBDIR1) \\\n+\t\tSHELL_PATH='$(SHELL_PATH_SQ)' PERL_PATH='$(PERL_PATH_SQ)'\n+\ttest -n \"$(ARTIFACTS_DIRECTORY)\"\n+\tmkdir -p \"$(ARTIFACTS_DIRECTORY)\"\n+\t$(TAR) czf \"$(ARTIFACTS_DIRECTORY)/artifacts.tar.gz\" $^ templates/blt/\n+.PHONY: artifacts-tar\n+\n htmldocs = git-htmldocs-$(GIT_VERSION)\n manpages = git-manpages-$(GIT_VERSION)\n .PHONY: dist-doc distclean\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex 480e841a85..c329b7218b 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -3,8 +3,8 @@ resources:\n fetchDepth: 1\n \n jobs:\n-- job: windows\n- displayName: Windows\n+- job: windows_build\n+ displayName: Windows Build\n condition: succeeded()\n pool: Hosted\n timeoutInMinutes: 240\n@@ -30,21 +30,84 @@ jobs:\n displayName: 'Download git-sdk-64-minimal'\n - powershell: |\n & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n- export DEVELOPER=1\n- export NO_PERL=1\n- export NO_SVN_TESTS=1\n- export GIT_TEST_SKIP_REBASE_P=1\n+ ci/make-test-artifacts.sh artifacts\n+ \"@\n+ if (!$?) { exit(1) }\n+ displayName: Build\n+ env:\n+ HOME: $(Build.SourcesDirectory)\n+ MSYSTEM: MINGW64\n+ DEVELOPER: 1\n+ NO_PERL: 1\n+ - task: PublishPipelineArtifact@0\n+ displayName: 'Publish Pipeline Artifact: test artifacts'\n+ inputs:\n+ artifactName: 'windows-artifacts'\n+ targetPath: '$(Build.SourcesDirectory)\\artifacts'\n+ - task: PublishPipelineArtifact@0\n+ displayName: 'Publish Pipeline Artifact: git-sdk-64-minimal'\n+ inputs:\n+ artifactName: 'git-sdk-64-minimal'\n+ targetPath: '$(Build.SourcesDirectory)\\git-sdk-64-minimal'\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'Unmount test-cache'\n+ condition: true\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+\n+- job: windows_test\n+ displayName: Windows Test\n+ dependsOn: windows_build\n+ condition: succeeded()\n+ pool: Hosted\n+ timeoutInMinutes: 240\n+ strategy:\n+ parallel: 10\n+ steps:\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n+ }\n+ displayName: 'Mount test-cache'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: DownloadPipelineArtifact@0\n+ displayName: 'Download Pipeline Artifact: test artifacts'\n+ inputs:\n+ artifactName: 'windows-artifacts'\n+ targetPath: '$(Build.SourcesDirectory)'\n+ - task: DownloadPipelineArtifact@0\n+ displayName: 'Download Pipeline Artifact: git-sdk-64-minimal'\n+ inputs:\n+ artifactName: 'git-sdk-64-minimal'\n+ targetPath: '$(Build.SourcesDirectory)\\git-sdk-64-minimal'\n+ - powershell: |\n+ & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n+ test -f artifacts.tar.gz || {\n+ echo No test artifacts found\\; skipping >&2\n+ exit 0\n+ }\n+ tar xf artifacts.tar.gz || exit 1\n+\n+ # Let Git ignore the SDK and the test-cache\n+ printf '%s\\n' /git-sdk-64-minimal/ /test-cache/ >>.git/info/exclude\n \n- ci/run-build-and-tests.sh || {\n+ ci/run-test-slice.sh `$SYSTEM_JOBPOSITIONINPHASE `$SYSTEM_TOTALJOBSINPHASE || {\n ci/print-test-failures.sh\n exit 1\n }\n \"@\n if (!$?) { exit(1) }\n- displayName: 'Build & Test'\n+ displayName: 'Test (parallel)'\n env:\n HOME: $(Build.SourcesDirectory)\n MSYSTEM: MINGW64\n+ NO_SVN_TESTS: 1\n+ GIT_TEST_SKIP_REBASE_P: 1\n - powershell: |\n if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\ndiff --git a/ci/make-test-artifacts.sh b/ci/make-test-artifacts.sh\nnew file mode 100755\nindex 0000000000..646967481f\n--- /dev/null\n+++ b/ci/make-test-artifacts.sh\n@@ -0,0 +1,12 @@\n+#!/bin/sh\n+#\n+# Build Git and store artifacts for testing\n+#\n+\n+mkdir -p \"$1\" # in case ci/lib.sh decides to quit early\n+\n+. ${0%/*}/lib.sh\n+\n+make artifacts-tar ARTIFACTS_DIRECTORY=\"$1\"\n+\n+check_unignored_build_artifacts\ndiff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh\nnew file mode 100755\nindex 0000000000..f8c2c3106a\n--- /dev/null\n+++ b/ci/run-test-slice.sh\n@@ -0,0 +1,17 @@\n+#!/bin/sh\n+#\n+# Test Git in parallel\n+#\n+\n+. ${0%/*}/lib.sh\n+\n+case \"$CI_OS_NAME\" in\n+windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n+*) ln -s \"$cache_dir/.prove\" t/.prove;;\n+esac\n+\n+make --quiet -C t T=\"$(cd t &&\n+\t./helper/test-tool path-utils slice-tests \"$1\" \"$2\" t[0-9]*.sh |\n+\ttr '\\n' ' ')\"\n+\n+check_unignored_build_artifacts\ndiff --git a/t/helper/test-path-utils.c b/t/helper/test-path-utils.c\nindex 6efde6f5ba..5d543ad21f 100644\n--- a/t/helper/test-path-utils.c\n+++ b/t/helper/test-path-utils.c\n@@ -177,6 +177,14 @@ static int is_dotgitmodules(const char *path)\n \treturn is_hfs_dotgitmodules(path) || is_ntfs_dotgitmodules(path);\n }\n \n+static int cmp_by_st_size(const void *a, const void *b)\n+{\n+\tintptr_t x = (intptr_t)((struct string_list_item *)a)->util;\n+\tintptr_t y = (intptr_t)((struct string_list_item *)b)->util;\n+\n+\treturn x > y ? -1 : (x < y ? +1 : 0);\n+}\n+\n int cmd__path_utils(int argc, const char **argv)\n {\n \tif (argc == 3 && !strcmp(argv[1], \"normalize_path_copy\")) {\n@@ -324,6 +332,29 @@ int cmd__path_utils(int argc, const char **argv)\n \t\treturn 0;\n \t}\n \n+\tif (argc > 5 && !strcmp(argv[1], \"slice-tests\")) {\n+\t\tint res = 0;\n+\t\tlong offset, stride, i;\n+\t\tstruct string_list list = STRING_LIST_INIT_NODUP;\n+\t\tstruct stat st;\n+\n+\t\toffset = strtol(argv[2], NULL, 10);\n+\t\tstride = strtol(argv[3], NULL, 10);\n+\t\tif (stride < 1)\n+\t\t\tstride = 1;\n+\t\tfor (i = 4; i < argc; i++)\n+\t\t\tif (stat(argv[i], &st))\n+\t\t\t\tres = error_errno(\"Cannot stat '%s'\", argv[i]);\n+\t\t\telse\n+\t\t\t\tstring_list_append(&list, argv[i])->util =\n+\t\t\t\t\t(void *)(intptr_t)st.st_size;\n+\t\tQSORT(list.items, list.nr, cmp_by_st_size);\n+\t\tfor (i = offset; i < list.nr; i+= stride)\n+\t\t\tprintf(\"%s\\n\", list.items[i].string);\n+\n+\t\treturn !!res;\n+\t}\n+\n \tfprintf(stderr, \"%s: unknown function name: %s\\n\", argv[0],\n \t\targv[1] ? argv[1] : \"(there was none)\");\n \treturn 1;\n-- \ngitgitgadget\n"},{"id":"367975","messageId":"4ec6cc83ef03a68cb0efacc530e8413fded83b95.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 18/21] t0061: workaround issues with --with-dashes and RUNTIME_PREFIX","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:36Z","receivedAt":"2019-01-29T14:19:45Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nWhen building Git with RUNTIME_PREFIX and starting a test helper from\nt/helper/, it fails to detect a system prefix. The reason is that the\nRUNTIME_PREFIX feature wants to use the location of the Git executable\nto determine where the support files can be found, e.g. system-wide Git\nconfig or the translations. This does not make any sense for the test\nhelpers, though, as they are distinctly not in a directory structure\nresembling the final installation location of Git.\n\nThat is the reason why the test helpers rely on environment variables to\nindicate the location of the needed support files, e.g.\nGIT_TEXTDOMAINDIR. If this information is missing, the output will\ncontain warnings like this one:\n\n\tRUNTIME_PREFIX requested, but prefix computation failed. [...]\n\nIn t0061, we did not expect that to happen, and it actually does not\nhappen in the regular case, because bin-wrappers/test-tool specifically\nsets GIT_TEXTDOMAINDIR (and as a consequence, nothing in test-tool needs\nto know anything about any runtime prefix).\n\nHowever, with --with-dashes, bin-wrappers/test-tool is no longer called,\nbut t/helper/test-tool is called directly instead.\n\nSo let's just ignore the RUNTIME_PREFIX warning.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t0061-run-command.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex 99a614bc7c..5a2d087bf0 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -166,7 +166,8 @@ test_trace () {\n \texpect=\"$1\"\n \tshift\n \tGIT_TRACE=1 test-tool run-command \"$@\" run-command true 2>&1 >/dev/null | \\\n-\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' >actual &&\n+\t\tsed -e 's/.* run_command: //' -e '/trace: .*/d' \\\n+\t\t\t-e '/RUNTIME_PREFIX requested/d' >actual &&\n \techo \"$expect true\" >expect &&\n \ttest_cmp expect actual\n }\n-- \ngitgitgadget\n\n"},{"id":"367976","messageId":"b6316e1f7dee45d576eb04cbde2f35f09e8e56c3.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 13/21] README: add a build badge (status of the Azure Pipelines build)","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:32Z","receivedAt":"2019-01-29T14:19:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nJust like so many other OSS projects, we now also have a build badge.\n\nSigned-off-by: Johannes Schindelin \n---\n README.md | 2 ++\n 1 file changed, 2 insertions(+)\n\ndiff --git a/README.md b/README.md\nindex f920a42fad..764c480c66 100644\n--- a/README.md\n+++ b/README.md\n@@ -1,3 +1,5 @@\n+[![Build Status](https://dev.azure.com/git/git/_apis/build/status/test-git.git)](https://dev.azure.com/git/git/_build/latest?definitionId=2)\n+\n Git - fast, scalable, distributed revision control system\n =========================================================\n \n-- \ngitgitgadget\n\n"},{"id":"367977","messageId":"1ab1d14e063969c86db2accf7340dcd5fcde27e5.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 11/21] ci: use git-sdk-64-minimal build artifact","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:31Z","receivedAt":"2019-01-29T14:19:47Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nInstead of a shallow fetch followed by a sparse checkout, we are\nbetter off by using a separate, dedicated Pipeline that bundles\nthe SDK as a build artifact, and then consuming that build artifact\nhere.\n\nIn fact, since this artifact will be used a lot, we spent substantial\ntime on figuring out a minimal subset of the Git for Windows SDK, just\nenough to build and test Git. The result is a size reduction from around\n1GB (compressed) to around 55MB (compressed). This also comes with the\nchange where we now call `usr\\bin\\bash.exe` directly, as `git-cmd.exe`\nis not included in the minimal SDK.\n\nThat reduces the time to initialize Git for Windows' SDK from anywhere\nbetween 2m30s-7m to a little over 1m.\n\nNote: in theory, we could also use the DownloadBuildArtifacts@0 task\nhere. However, restricted permissions that are in effect when building\nfrom forks would let this fail for PR builds, defeating the whole\npurpose of the Azure Pipelines support for git.git.\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 44 +++++++++-----------------------------------\n 1 file changed, 9 insertions(+), 35 deletions(-)\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex 0f7b2125a1..480e841a85 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -18,44 +18,18 @@ jobs:\n env:\n GITFILESHAREPWD: $(gitfileshare.pwd)\n - powershell: |\n- # Helper to check the error level of the latest command (exit with error when appropriate)\n- function c() { if (!$?) { exit(1) } }\n-\n- # Add build agent's MinGit to PATH\n- $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n-\n- # Helper to initialize (or update) a Git worktree\n- function init ($path, $url, $set_origin) {\n- if (Test-Path $path) {\n- cd $path; c\n- if (Test-Path .git) {\n- & git init; c\n- } else {\n- & git status\n- }\n- } else {\n- & git init $path; c\n- cd $path; c\n- }\n- & git config core.autocrlf false; c\n- & git config core.untrackedCache true; c\n- if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n- & git remote add origin $url; c\n- }\n- & git fetch --depth=1 $url master; c\n- & git reset --hard FETCH_HEAD; c\n- & git clean -df; c\n- }\n-\n- # Initialize Git for Windows' SDK\n- $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n- init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+ $urlbase = \"https://dev.azure.com/git-for-windows/git/_apis/build/builds\"\n+ $id = ((Invoke-WebRequest -UseBasicParsing \"${urlbase}?definitions=22&statusFilter=completed&resultFilter=succeeded&`$top=1\").content | ConvertFrom-JSON).value[0].id\n+ $downloadUrl = ((Invoke-WebRequest -UseBasicParsing \"${urlbase}/$id/artifacts\").content | ConvertFrom-JSON).value[1].resource.downloadUrl\n+ (New-Object Net.WebClient).DownloadFile($downloadUrl,\"git-sdk-64-minimal.zip\")\n+ Expand-Archive git-sdk-64-minimal.zip -DestinationPath . -Force\n+ Remove-Item git-sdk-64-minimal.zip\n \n # Let Git ignore the SDK and the test-cache\n- \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n- displayName: 'Initialize the Git for Windows SDK'\n+ \"/git-sdk-64-minimal/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n+ displayName: 'Download git-sdk-64-minimal'\n - powershell: |\n- & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ & git-sdk-64-minimal\\usr\\bin\\bash.exe -lc @\"\n export DEVELOPER=1\n export NO_PERL=1\n export NO_SVN_TESTS=1\n-- \ngitgitgadget\n\n"},{"id":"367978","messageId":"4c78085af7b41dc66a2757dfad9dd1ab329cff5b.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 08/21] ci/lib.sh: add support for Azure Pipelines","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:28Z","receivedAt":"2019-01-29T14:19:48Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis patch introduces a conditional arm that defines some environment\nvariables and a function that displays the URL given the job id (to\nidentify previous runs for known-good trees).\n\nBecause Azure Pipeline's macOS agents already have git-lfs and gettext\ninstalled, we can leave `BREW_INSTALL_PACKAGES` empty (unlike in\nTravis' case).\n\nNote: this patch does not introduce an Azure Pipelines definition yet;\nThat is left for the next patch.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/lib.sh | 25 +++++++++++++++++++++++++\n ci/print-test-failures.sh | 5 +++++\n 2 files changed, 30 insertions(+)\n\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 32a28fd209..5505776876 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -102,6 +102,31 @@ then\n \texport GIT_PROVE_OPTS=\"--timer --jobs 3 --state=failed,slow,save\"\n \texport GIT_TEST_OPTS=\"--verbose-log -x --immediate\"\n \texport MAKEFLAGS=\"--jobs=2\"\n+elif test -n \"$SYSTEM_COLLECTIONURI\" || test -n \"$SYSTEM_TASKDEFINITIONSURI\"\n+then\n+\tCI_TYPE=azure-pipelines\n+\t# We are running in Azure Pipelines\n+\tCI_BRANCH=\"$BUILD_SOURCEBRANCH\"\n+\tCI_COMMIT=\"$BUILD_SOURCEVERSION\"\n+\tCI_JOB_ID=\"$BUILD_BUILDID\"\n+\tCI_JOB_NUMBER=\"$BUILD_BUILDNUMBER\"\n+\tCI_OS_NAME=\"$(echo \"$AGENT_OS\" | tr A-Z a-z)\"\n+\ttest darwin != \"$CI_OS_NAME\" || CI_OS_NAME=osx\n+\tCI_REPO_SLUG=\"$(expr \"$BUILD_REPOSITORY_URI\" : '.*/\\([^/]*/[^/]*\\)$')\"\n+\tCC=\"${CC:-gcc}\"\n+\n+\t# use a subdirectory of the cache dir (because the file share is shared\n+\t# among *all* phases)\n+\tcache_dir=\"$HOME/test-cache/$SYSTEM_PHASENAME\"\n+\n+\turl_for_job_id () {\n+\t\techo \"$SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$1\"\n+\t}\n+\n+\tBREW_INSTALL_PACKAGES=\n+\texport GIT_PROVE_OPTS=\"--timer --jobs 10 --state=failed,slow,save\"\n+\texport GIT_TEST_OPTS=\"--verbose-log -x --write-junit-xml\"\n+\texport MAKEFLAGS=\"--jobs=10\"\n else\n \techo \"Could not identify CI type\" >&2\n \texit 1\ndiff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh\nindex cf321b474d..e688a26f0d 100755\n--- a/ci/print-test-failures.sh\n+++ b/ci/print-test-failures.sh\n@@ -41,6 +41,11 @@ do\n \t\tcase \"$CI_TYPE\" in\n \t\ttravis)\n \t\t\t;;\n+\t\tazure-pipelines)\n+\t\t\tmkdir -p failed-test-artifacts\n+\t\t\tmv \"$trash_dir\" failed-test-artifacts\n+\t\t\tcontinue\n+\t\t\t;;\n \t\t*)\n \t\t\techo \"Unhandled CI type: $CI_TYPE\" >&2\n \t\t\texit 1\n-- \ngitgitgadget\n\n"},{"id":"367979","messageId":"e9a869df573ac8135561110fe2c366df7597ffc4.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 07/21] tests: optionally write results as JUnit-style .xml","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:27Z","receivedAt":"2019-01-29T14:19:49Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nThis will come in handy when publishing the results of Git's test suite\nduring an automated Azure DevOps run.\n\nNote: we need to make extra sure that invalid UTF-8 encoding is turned\ninto valid UTF-8 (using the Replacement Character, \\uFFFD) because\nt9902's trace contains such invalid byte sequences, and the task in the\nAzure Pipeline that uploads the test results would refuse to do anything\nif it was asked to parse an .xml file with invalid UTF-8 in it.\n\nSigned-off-by: Johannes Schindelin \n---\n Makefile | 1 +\n t/.gitignore | 1 +\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/helper/test-xml-encode.c | 80 +++++++++++++++++++++++++++++++++\n t/test-lib.sh | 91 ++++++++++++++++++++++++++++++++++++++\n 6 files changed, 175 insertions(+)\n create mode 100644 t/helper/test-xml-encode.c\n\ndiff --git a/Makefile b/Makefile\nindex 1a44c811aa..044b4f77bd 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -754,6 +754,7 @@ TEST_BUILTINS_OBJS += test-submodule-config.o\n TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o\n TEST_BUILTINS_OBJS += test-subprocess.o\n TEST_BUILTINS_OBJS += test-urlmatch-normalization.o\n+TEST_BUILTINS_OBJS += test-xml-encode.o\n TEST_BUILTINS_OBJS += test-wildmatch.o\n TEST_BUILTINS_OBJS += test-windows-named-pipe.o\n TEST_BUILTINS_OBJS += test-write-cache.o\ndiff --git a/t/.gitignore b/t/.gitignore\nindex 348715f0e4..91cf5772fe 100644\n--- a/t/.gitignore\n+++ b/t/.gitignore\n@@ -2,3 +2,4 @@\n /test-results\n /.prove\n /chainlinttmp\n+/out/\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex bfb195b1a8..4b4b397d93 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -49,6 +49,7 @@ static struct test_cmd cmds[] = {\n \t{ \"submodule-nested-repo-config\", cmd__submodule_nested_repo_config },\n \t{ \"subprocess\", cmd__subprocess },\n \t{ \"urlmatch-normalization\", cmd__urlmatch_normalization },\n+\t{ \"xml-encode\", cmd__xml_encode },\n \t{ \"wildmatch\", cmd__wildmatch },\n #ifdef GIT_WINDOWS_NATIVE\n \t{ \"windows-named-pipe\", cmd__windows_named_pipe },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 042f12464b..c0ab65e370 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -45,6 +45,7 @@ int cmd__submodule_config(int argc, const char **argv);\n int cmd__submodule_nested_repo_config(int argc, const char **argv);\n int cmd__subprocess(int argc, const char **argv);\n int cmd__urlmatch_normalization(int argc, const char **argv);\n+int cmd__xml_encode(int argc, const char **argv);\n int cmd__wildmatch(int argc, const char **argv);\n #ifdef GIT_WINDOWS_NATIVE\n int cmd__windows_named_pipe(int argc, const char **argv);\ndiff --git a/t/helper/test-xml-encode.c b/t/helper/test-xml-encode.c\nnew file mode 100644\nindex 0000000000..a648bbd961\n--- /dev/null\n+++ b/t/helper/test-xml-encode.c\n@@ -0,0 +1,80 @@\n+#include \"test-tool.h\"\n+\n+static const char *utf8_replace_character = \"�\";\n+\n+/*\n+ * Encodes (possibly incorrect) UTF-8 on to , to be embedded\n+ * in an XML file.\n+ */\n+int cmd__xml_encode(int argc, const char **argv)\n+{\n+\tunsigned char buf[1024], tmp[4], *tmp2 = NULL;\n+\tssize_t cur = 0, len = 1, remaining = 0;\n+\tunsigned char ch;\n+\n+\tfor (;;) {\n+\t\tif (++cur == len) {\n+\t\t\tlen = xread(0, buf, sizeof(buf));\n+\t\t\tif (!len)\n+\t\t\t\treturn 0;\n+\t\t\tif (len < 0)\n+\t\t\t\tdie_errno(\"Could not read \");\n+\t\t\tcur = 0;\n+\t\t}\n+\t\tch = buf[cur];\n+\n+\t\tif (tmp2) {\n+\t\t\tif ((ch & 0xc0) != 0x80) {\n+\t\t\t\tfputs(utf8_replace_character, stdout);\n+\t\t\t\ttmp2 = NULL;\n+\t\t\t\tcur--;\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\t*tmp2 = ch;\n+\t\t\ttmp2++;\n+\t\t\tif (--remaining == 0) {\n+\t\t\t\tfwrite(tmp, tmp2 - tmp, 1, stdout);\n+\t\t\t\ttmp2 = NULL;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (!(ch & 0x80)) {\n+\t\t\t/* 0xxxxxxx */\n+\t\t\tif (ch == '&')\n+\t\t\t\tfputs(\"&\", stdout);\n+\t\t\telse if (ch == '\\'')\n+\t\t\t\tfputs(\"'\", stdout);\n+\t\t\telse if (ch == '\"')\n+\t\t\t\tfputs(\""\", stdout);\n+\t\t\telse if (ch == '<')\n+\t\t\t\tfputs(\"<\", stdout);\n+\t\t\telse if (ch == '>')\n+\t\t\t\tfputs(\">\", stdout);\n+\t\t\telse if (ch >= 0x20)\n+\t\t\t\tfputc(ch, stdout);\n+\t\t\telse if (ch == 0x09 || ch == 0x0a || ch == 0x0d)\n+\t\t\t\tfprintf(stdout, \"&#x%02x;\", ch);\n+\t\t\telse\n+\t\t\t\tfputs(utf8_replace_character, stdout);\n+\t\t} else if ((ch & 0xe0) == 0xc0) {\n+\t\t\t/* 110XXXXx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 1;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else if ((ch & 0xf0) == 0xe0) {\n+\t\t\t/* 1110XXXX 10Xxxxxx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 2;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else if ((ch & 0xf8) == 0xf0) {\n+\t\t\t/* 11110XXX 10XXxxxx 10xxxxxx 10xxxxxx */\n+\t\t\ttmp[0] = ch;\n+\t\t\tremaining = 3;\n+\t\t\ttmp2 = tmp + 1;\n+\t\t} else\n+\t\t\tfputs(utf8_replace_character, stdout);\n+\t}\n+\n+\treturn 0;\n+}\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex a1abb1177a..a3b2166cb5 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -139,6 +139,9 @@ do\n \t\tverbose_log=t\n \t\ttee=t\n \t\t;;\n+\t--write-junit-xml)\n+\t\twrite_junit_xml=t\n+\t\t;;\n \t--stress)\n \t\tstress=t ;;\n \t--stress=*)\n@@ -622,11 +625,24 @@ trap 'exit $?' INT TERM HUP\n # the test_expect_* functions instead.\n \n test_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$*\"\n+\tfi\n \ttest_success=$(($test_success + 1))\n \tsay_color \"\" \"ok $test_count - $@\"\n }\n \n test_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_insert=\"\"\n+\t\tjunit_insert=\"$junit_insert $(xml_attr_encode \\\n+\t\t\t\"$(printf '%s\\n' \"$@\" | sed 1d)\")\"\n+\t\tjunit_insert=\"$junit_insert\"\n+\t\twrite_junit_xml_testcase \"$1\" \" $junit_insert\"\n+\tfi\n \ttest_failure=$(($test_failure + 1))\n \tsay_color error \"not ok $test_count - $1\"\n \tshift\n@@ -635,11 +651,19 @@ test_failure_ () {\n }\n \n test_known_broken_ok_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (breakage fixed)\"\n+\tfi\n \ttest_fixed=$(($test_fixed+1))\n \tsay_color error \"ok $test_count - $@ # TODO known breakage vanished\"\n }\n \n test_known_broken_failure_ () {\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\twrite_junit_xml_testcase \"$* (known breakage)\"\n+\tfi\n \ttest_broken=$(($test_broken+1))\n \tsay_color warn \"not ok $test_count - $@ # TODO known breakage\"\n }\n@@ -897,6 +921,10 @@ test_start_ () {\n \ttest_count=$(($test_count+1))\n \tmaybe_setup_verbose\n \tmaybe_setup_valgrind\n+\tif test -n \"$write_junit_xml\"\n+\tthen\n+\t\tjunit_start=$(test-tool date getnanos)\n+\tfi\n }\n \n test_finish_ () {\n@@ -934,6 +962,13 @@ test_skip () {\n \n \tcase \"$to_skip\" in\n \tt)\n+\t\tif test -n \"$write_junit_xml\"\n+\t\tthen\n+\t\t\tmessage=\"$(xml_attr_encode \"$skipped_reason\")\"\n+\t\t\twrite_junit_xml_testcase \"$1\" \\\n+\t\t\t\t\" \"\n+\t\tfi\n+\n \t\tsay_color skip >&3 \"skipping test: $@\"\n \t\tsay_color skip \"ok $test_count # skip $1 ($skipped_reason)\"\n \t\t: true\n@@ -949,9 +984,51 @@ test_at_end_hook_ () {\n \t:\n }\n \n+write_junit_xml () {\n+\tcase \"$1\" in\n+\t--truncate)\n+\t\t>\"$junit_xml_path\"\n+\t\tjunit_have_testcase=\n+\t\tshift\n+\t\t;;\n+\tesac\n+\tprintf '%s\\n' \"$@\" >>\"$junit_xml_path\"\n+}\n+\n+xml_attr_encode () {\n+\tprintf '%s\\n' \"$@\" | test-tool xml-encode\n+}\n+\n+write_junit_xml_testcase () {\n+\tjunit_attrs=\"name=\\\"$(xml_attr_encode \"$this_test.$test_count $1\")\\\"\"\n+\tshift\n+\tjunit_attrs=\"$junit_attrs classname=\\\"$this_test\\\"\"\n+\tjunit_attrs=\"$junit_attrs time=\\\"$(test-tool \\\n+\t\tdate getnanos $junit_start)\\\"\"\n+\twrite_junit_xml \"$(printf '%s\\n' \\\n+\t\t\" \" \"$@\" \" \")\"\n+\tjunit_have_testcase=t\n+}\n+\n test_done () {\n \tGIT_EXIT_OK=t\n \n+\tif test -n \"$write_junit_xml\" && test -n \"$junit_xml_path\"\n+\tthen\n+\t\ttest -n \"$junit_have_testcase\" || {\n+\t\t\tjunit_start=$(test-tool date getnanos)\n+\t\t\twrite_junit_xml_testcase \"all tests skipped\"\n+\t\t}\n+\n+\t\t# adjust the overall time\n+\t\tjunit_time=$(test-tool date getnanos $junit_suite_start)\n+\t\tsed \"s/]*/& time=\\\"$junit_time\\\"/\" \\\n+\t\t\t<\"$junit_xml_path\" >\"$junit_xml_path.new\"\n+\t\tmv \"$junit_xml_path.new\" \"$junit_xml_path\"\n+\n+\t\twrite_junit_xml \" \" \"\"\n+\tfi\n+\n \tif test -z \"$HARNESS_ACTIVE\"\n \tthen\n \t\tmkdir -p \"$TEST_RESULTS_DIR\"\n@@ -1178,6 +1255,7 @@ then\n else\n \tmkdir -p \"$TRASH_DIRECTORY\"\n fi\n+\n # Use -P to resolve symlinks in our working directory so that the cwd\n # in subprocesses like git equals our $PWD (for pathname comparisons).\n cd -P \"$TRASH_DIRECTORY\" || exit 1\n@@ -1191,6 +1269,19 @@ then\n \ttest_done\n fi\n \n+if test -n \"$write_junit_xml\"\n+then\n+\tjunit_xml_dir=\"$TEST_OUTPUT_DIRECTORY/out\"\n+\tmkdir -p \"$junit_xml_dir\"\n+\tjunit_xml_base=${0##*/}\n+\tjunit_xml_path=\"$junit_xml_dir/TEST-${junit_xml_base%.sh}.xml\"\n+\tjunit_attrs=\"name=\\\"${junit_xml_base%.sh}\\\"\"\n+\tjunit_attrs=\"$junit_attrs timestamp=\\\"$(TZ=UTC \\\n+\t\tdate +%Y-%m-%dT%H:%M:%S)\\\"\"\n+\twrite_junit_xml --truncate \"\" \" \"\n+\tjunit_suite_start=$(test-tool date getnanos)\n+fi\n+\n # Provide an implementation of the 'yes' utility\n yes () {\n \tif test $# = 0\n-- \ngitgitgadget\n\n"},{"id":"367980","messageId":"d5651317817a44d8f8b565c2cc14f59dfc432916.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 10/21] ci: add a Windows job to the Azure Pipelines definition","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:30Z","receivedAt":"2019-01-29T14:19:50Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nPreviously, we did not have robust support for Windows in our CI\ndefinition, simply because Travis cannot accommodate our needs (even\nafter Travis added experimental Windows support very recently, it takes\nlonger than Travis' 50 minute timeout to build Git and run the test\nsuite on Windows). Instead, we used a hack that started a dedicated\nAzure Pipeline from Travis and waited for the output, often timing out\n(which is quite fragile, as we found out).\n\nWith this commit, we finally have first-class support for Windows in our\nCI definition (in the Azure Pipelines one, that is).\n\nDue to our reliance on Unix shell scripting in the test suite, combined\nwith the challenges on executing such scripts on Windows, the Windows\njob currently takes a whopping ~1h20m to complete. Which is *far* longer\nthan the next-longest job takes (linux-gcc, ~35m).\n\nNow, Azure Pipelines's free tier for open source projects (such as Git)\noffers up to 10 concurrent jobs for free, meaning that the overall run\ntime will be dominated by the slowest job(s).\n\nTherefore, it makes sense to start the Windows job first, to minimize\nthe time the entire build takes from start to end (which is now pretty\nsafely the run time of the Windows job).\n\nSigned-off-by: Johannes Schindelin \n---\n azure-pipelines.yml | 91 +++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 91 insertions(+)\n\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex 8cdef105c6..0f7b2125a1 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -3,6 +3,97 @@ resources:\n fetchDepth: 1\n \n jobs:\n+- job: windows\n+ displayName: Windows\n+ condition: succeeded()\n+ pool: Hosted\n+ timeoutInMinutes: 240\n+ steps:\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ net use s: \\\\gitfileshare.file.core.windows.net\\test-cache \"$GITFILESHAREPWD\" /user:AZURE\\gitfileshare /persistent:no\n+ cmd /c mklink /d \"$(Build.SourcesDirectory)\\test-cache\" S:\\\n+ }\n+ displayName: 'Mount test-cache'\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - powershell: |\n+ # Helper to check the error level of the latest command (exit with error when appropriate)\n+ function c() { if (!$?) { exit(1) } }\n+\n+ # Add build agent's MinGit to PATH\n+ $env:PATH = $env:AGENT_HOMEDIRECTORY +\"\\externals\\\\git\\cmd;\" +$env:PATH\n+\n+ # Helper to initialize (or update) a Git worktree\n+ function init ($path, $url, $set_origin) {\n+ if (Test-Path $path) {\n+ cd $path; c\n+ if (Test-Path .git) {\n+ & git init; c\n+ } else {\n+ & git status\n+ }\n+ } else {\n+ & git init $path; c\n+ cd $path; c\n+ }\n+ & git config core.autocrlf false; c\n+ & git config core.untrackedCache true; c\n+ if (($set_origin -ne 0) -and !(git config remote.origin.url)) {\n+ & git remote add origin $url; c\n+ }\n+ & git fetch --depth=1 $url master; c\n+ & git reset --hard FETCH_HEAD; c\n+ & git clean -df; c\n+ }\n+\n+ # Initialize Git for Windows' SDK\n+ $sdk_path = \"$(Build.SourcesDirectory)\\git-sdk-64\"\n+ init \"$sdk_path\" \"https://dev.azure.com/git-for-windows/git-sdk-64/_git/git-sdk-64\" 0\n+\n+ # Let Git ignore the SDK and the test-cache\n+ \"/git-sdk-64/`n/test-cache/`n\" | Out-File -NoNewLine -Encoding ascii -Append \"$(Build.SourcesDirectory)\\.git\\info\\exclude\"\n+ displayName: 'Initialize the Git for Windows SDK'\n+ - powershell: |\n+ & \"git-sdk-64\\git-cmd.exe\" --command=usr\\\\bin\\\\bash.exe -lc @\"\n+ export DEVELOPER=1\n+ export NO_PERL=1\n+ export NO_SVN_TESTS=1\n+ export GIT_TEST_SKIP_REBASE_P=1\n+\n+ ci/run-build-and-tests.sh || {\n+ ci/print-test-failures.sh\n+ exit 1\n+ }\n+ \"@\n+ if (!$?) { exit(1) }\n+ displayName: 'Build & Test'\n+ env:\n+ HOME: $(Build.SourcesDirectory)\n+ MSYSTEM: MINGW64\n+ - powershell: |\n+ if (\"$GITFILESHAREPWD\" -ne \"\" -and \"$GITFILESHAREPWD\" -ne \"`$`(gitfileshare.pwd)\") {\n+ cmd /c rmdir \"$(Build.SourcesDirectory)\\test-cache\"\n+ }\n+ displayName: 'Unmount test-cache'\n+ condition: true\n+ env:\n+ GITFILESHAREPWD: $(gitfileshare.pwd)\n+ - task: PublishTestResults@2\n+ displayName: 'Publish Test Results **/TEST-*.xml'\n+ inputs:\n+ mergeTestResults: true\n+ testRunTitle: 'windows'\n+ platform: Windows\n+ publishRunAttachments: false\n+ condition: succeededOrFailed()\n+ - task: PublishBuildArtifacts@1\n+ displayName: 'Publish trash directories of failed tests'\n+ condition: failed()\n+ inputs:\n+ PathtoPublish: t/failed-test-artifacts\n+ ArtifactName: failed-test-artifacts\n+\n - job: linux_clang\n displayName: linux-clang\n condition: succeeded()\n-- \ngitgitgadget\n\n"},{"id":"367981","messageId":"681f8e65b4d3cf82a5920840aa8dc88bde812964.1548771561.git.gitgitgadget@gmail.com","threadId":"49270","inReplyTo":"pull.31.v6.git.gitgitgadget@gmail.com","subject":"[PATCH v6 05/21] ci: use a junction on Windows instead of a symlink","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-01-29T14:19:26Z","receivedAt":"2019-01-29T14:19:52Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin \n\nSymbolic links are still not quite as easy to use on Windows as on Linux\n(for example, on versions older than Windows 10, only administrators can\ncreate symlinks, and on Windows 10 you still need to be in developer\nmode for regular users to have permission), but NTFS junctions can give\nus a way out.\n\nSigned-off-by: Johannes Schindelin \n---\n ci/run-build-and-tests.sh | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex 80d72d120f..74d838ea01 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -5,7 +5,10 @@\n \n . ${0%/*}/lib.sh\n \n-ln -s \"$cache_dir/.prove\" t/.prove\n+case \"$CI_OS_NAME\" in\n+windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n+*) ln -s \"$cache_dir/.prove\" t/.prove;;\n+esac\n \n make\n make --quiet test\n-- \ngitgitgadget\n\n"},{"id":"418685","messageId":"87zgzbgp1i.fsf@evledraar.gmail.com","threadId":"49270","inReplyTo":"991b41afa4a83a73f59a72504d269d64d12ecf8f.1548771561.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v6 16/21] mingw: try to work around issues with the test cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-10T12:40:25Z","receivedAt":"2021-03-10T12:41:27Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jan 29 2019, Johannes Schindelin via GitGitGadget wrote:\n\n> From: Johannes Schindelin \n>\n> It seems that every once in a while in the Git for Windows SDK, there\n> are some transient file locking issues preventing the test clean up to\n> delete the trash directory. Let's be gentle and try again five seconds\n> later, and only error out if it still fails the second time.\n>\n> This change helps Windows, and does not hurt any other platform\n> (normally, it is highly unlikely that said deletion fails, and if it\n> does, normally it will fail again even 5 seconds later).\n>\n> Signed-off-by: Johannes Schindelin \n> ---\n> t/test-lib.sh | 6 +++++-\n> 1 file changed, 5 insertions(+), 1 deletion(-)\n>\n> diff --git a/t/test-lib.sh b/t/test-lib.sh\n> index f31a1c8f79..9c0ca5effb 100644\n> --- a/t/test-lib.sh\n> +++ b/t/test-lib.sh\n> @@ -1104,7 +1104,11 @@ test_done () {\n> \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n> \n> \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n> -\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n> +\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n> +\t\t\t\t# try again in a bit\n> +\t\t\t\tsleep 5;\n> +\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n> +\t\t\t} ||\n> \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n> \t\tfi\n> \t\ttest_at_end_hook_\n\nI saw this sleep while reading some test-lib.sh code, doesn't this break\ndf4c0d1a79 (test-lib: abort when can't remove trash directory,\n2017-04-20) for non-Windows platforms?\n\nYour CL for v3 suggests this was only encountered in Azure VMs:\nhttps://lore.kernel.org/git/pull.31.v3.git.gitgitgadget@gmail.com/\n\nAside from this obscure issue, wouldn't it make more sense to have some\noptional \"I'm under CI\" flag to skip the teardown one test at a time as\nwe're probably about to shut off the transitory VM soon?\n\nI skip some tests, but the test suite creates ~950MB of trash for\nme. Maybe cheaper for some to just keep that around and have it all\nremoved at the end.\n"},{"id":"419707","messageId":"nycvar.QRO.7.76.6.2103191515430.57@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"87zgzbgp1i.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 16/21] mingw: try to work around issues with the test cleanup","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-03-19T14:20:25Z","receivedAt":"2021-03-19T14:21:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nthanks for digging out this old thread, I really could have done never\nthinking about it again!\n\nOn Wed, 10 Mar 2021, Ævar Arnfjörð Bjarmason wrote:\n\n>\n> On Tue, Jan 29 2019, Johannes Schindelin via GitGitGadget wrote:\n>\n> > From: Johannes Schindelin \n> >\n> > It seems that every once in a while in the Git for Windows SDK, there\n> > are some transient file locking issues preventing the test clean up to\n> > delete the trash directory. Let's be gentle and try again five seconds\n> > later, and only error out if it still fails the second time.\n> >\n> > This change helps Windows, and does not hurt any other platform\n> > (normally, it is highly unlikely that said deletion fails, and if it\n> > does, normally it will fail again even 5 seconds later).\n> >\n> > Signed-off-by: Johannes Schindelin \n> > ---\n> > t/test-lib.sh | 6 +++++-\n> > 1 file changed, 5 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> > index f31a1c8f79..9c0ca5effb 100644\n> > --- a/t/test-lib.sh\n> > +++ b/t/test-lib.sh\n> > @@ -1104,7 +1104,11 @@ test_done () {\n> > \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n> >\n> > \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n> > -\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n> > +\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n> > +\t\t\t\t# try again in a bit\n> > +\t\t\t\tsleep 5;\n> > +\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n> > +\t\t\t} ||\n> > \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n> > \t\tfi\n> > \t\ttest_at_end_hook_\n>\n> I saw this sleep while reading some test-lib.sh code, doesn't this break\n> df4c0d1a79 (test-lib: abort when can't remove trash directory,\n> 2017-04-20) for non-Windows platforms?\n\nIt does not really break it, it just delays the inevitable failure.\n\n> Your CL for v3 suggests this was only encountered in Azure VMs:\n> https://lore.kernel.org/git/pull.31.v3.git.gitgitgadget@gmail.com/\n\nIf by \"CL\" you refer to the cover letter, then I might have made it sound\nas if it was only encountered in the Azure Pipelines agents. I vaguely\nseem to remember seeing something like this quite often on my personal\nmachine, too, though. Most likely Microsoft Defender going a little wild.\n\n> Aside from this obscure issue, wouldn't it make more sense to have some\n> optional \"I'm under CI\" flag to skip the teardown one test at a time as\n> we're probably about to shut off the transitory VM soon?\n\nNo, I'm not under CI, and I did encounter these issues. And they abruptly\nstopped with the patch you apparently still want to discuss ;-)\n\n> I skip some tests, but the test suite creates ~950MB of trash for\n> me. Maybe cheaper for some to just keep that around and have it all\n> removed at the end.\n\nI don't understand this statement. Or was there a question in it that\nyou'd like me to answer?\n\nCiao,\nJohannes\n"},{"id":"419776","messageId":"87a6qxzocs.fsf@evledraar.gmail.com","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.2103191515430.57@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v6 16/21] mingw: try to work around issues with the test cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-20T18:12:03Z","receivedAt":"2021-03-20T18:13:11Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Fri, Mar 19 2021, Johannes Schindelin wrote:\n\n> Hi Ævar,\n>\n> thanks for digging out this old thread, I really could have done never\n> thinking about it again!\n\nI didn't think I'd dug up something so traumatic :)\n\n> On Wed, 10 Mar 2021, Ævar Arnfjörð Bjarmason wrote:\n>\n>>\n>> On Tue, Jan 29 2019, Johannes Schindelin via GitGitGadget wrote:\n>>\n>> > From: Johannes Schindelin \n>> >\n>> > It seems that every once in a while in the Git for Windows SDK, there\n>> > are some transient file locking issues preventing the test clean up to\n>> > delete the trash directory. Let's be gentle and try again five seconds\n>> > later, and only error out if it still fails the second time.\n>> >\n>> > This change helps Windows, and does not hurt any other platform\n>> > (normally, it is highly unlikely that said deletion fails, and if it\n>> > does, normally it will fail again even 5 seconds later).\n>> >\n>> > Signed-off-by: Johannes Schindelin \n>> > ---\n>> > t/test-lib.sh | 6 +++++-\n>> > 1 file changed, 5 insertions(+), 1 deletion(-)\n>> >\n>> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n>> > index f31a1c8f79..9c0ca5effb 100644\n>> > --- a/t/test-lib.sh\n>> > +++ b/t/test-lib.sh\n>> > @@ -1104,7 +1104,11 @@ test_done () {\n>> > \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n>> >\n>> > \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n>> > -\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n>> > +\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n>> > +\t\t\t\t# try again in a bit\n>> > +\t\t\t\tsleep 5;\n>> > +\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n>> > +\t\t\t} ||\n>> > \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n>> > \t\tfi\n>> > \t\ttest_at_end_hook_\n>>\n>> I saw this sleep while reading some test-lib.sh code, doesn't this break\n>> df4c0d1a79 (test-lib: abort when can't remove trash directory,\n>> 2017-04-20) for non-Windows platforms?\n>\n> It does not really break it, it just delays the inevitable failure.\n>\n>> Your CL for v3 suggests this was only encountered in Azure VMs:\n>> https://lore.kernel.org/git/pull.31.v3.git.gitgitgadget@gmail.com/\n>\n> If by \"CL\" you refer to the cover letter, then I might have made it sound\n> as if it was only encountered in the Azure Pipelines agents. I vaguely\n\nYes, the cover letter.\n\n> seem to remember seeing something like this quite often on my personal\n> machine, too, though. Most likely Microsoft Defender going a little wild.\n>\n>> Aside from this obscure issue, wouldn't it make more sense to have some\n>> optional \"I'm under CI\" flag to skip the teardown one test at a time as\n>> we're probably about to shut off the transitory VM soon?\n>\n> No, I'm not under CI, and I did encounter these issues. And they abruptly\n> stopped with the patch you apparently still want to discuss ;-)\n>\n>> I skip some tests, but the test suite creates ~950MB of trash for\n>> me. Maybe cheaper for some to just keep that around and have it all\n>> removed at the end.\n>\n> I don't understand this statement. Or was there a question in it that\n> you'd like me to answer?\n\nIf the fix was purely for Azure's CI setup I was suggesting that a\nbetter solution might be to keep the trash around until the end. But\nthat's not the only setup as you note here, so let's discard that\nsuggestion.\n\nIn any case, your patch clearly undoes whatever canary for gc issues\ndf4c0d1a792 was trying to put into the test-lib, but didn't say so in\nits commit message.\n\nSo I figured it was something that was missed at the time, and that I\nshould send a quick E-Mail to both authors to see if anyone cared, maybe\nnobody does.\n\nIt's just something I ran into while reviewing test-lib.sh for some\nunrelated changes I was making...\n"},{"id":"419779","messageId":"xmqqmtuxk2m1.fsf@gitster.g","threadId":"49270","inReplyTo":"87a6qxzocs.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 16/21] mingw: try to work around issues with the test cleanup","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-03-20T20:10:46Z","receivedAt":"2021-03-20T20:11:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> In any case, your patch clearly undoes whatever canary for gc issues\n> df4c0d1a792 was trying to put into the test-lib, but didn't say so in\n> its commit message.\n>\n> So I figured it was something that was missed at the time, and that I\n> should send a quick E-Mail to both authors to see if anyone cared, maybe\n> nobody does.\n>\n> It's just something I ran into while reviewing test-lib.sh for some\n> unrelated changes I was making...\n\nGood eyes.\n\nI am not quite sure if we are better off catching a transitory\nfailure, which could go away when retried in 5 seconds, to remove\nthe test directory (in which case, the change in question is a clear\nregression from what df4c0d1a (test-lib: abort when can't remove\ntrash directory, 2017-04-20) intended to catch), or we only want to\ncatch permanent failures (in which case the patch in question is a\nclear improvement).\n\nIn either way, the decision and the rationale behind it should be in\nthe log message.\n\nThanks.\n"},{"id":"420117","messageId":"nycvar.QRO.7.76.6.2103241257580.50@tvgsbejvaqbjf.bet","threadId":"49270","inReplyTo":"87a6qxzocs.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 16/21] mingw: try to work around issues with the test cleanup","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-03-24T12:01:49Z","receivedAt":"2021-03-24T13:50:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Ævar,\n\nOn Sat, 20 Mar 2021, Ævar Arnfjörð Bjarmason wrote:\n\n> On Fri, Mar 19 2021, Johannes Schindelin wrote:\n>\n> >> On Tue, Jan 29 2019, Johannes Schindelin via GitGitGadget wrote:\n> >>\n> >> > From: Johannes Schindelin \n> >> >\n> >> > It seems that every once in a while in the Git for Windows SDK, there\n> >> > are some transient file locking issues preventing the test clean up to\n> >> > delete the trash directory. Let's be gentle and try again five seconds\n> >> > later, and only error out if it still fails the second time.\n> >> >\n> >> > This change helps Windows, and does not hurt any other platform\n> >> > (normally, it is highly unlikely that said deletion fails, and if it\n> >> > does, normally it will fail again even 5 seconds later).\n> >> >\n> >> > Signed-off-by: Johannes Schindelin \n> >> > ---\n> >> > t/test-lib.sh | 6 +++++-\n> >> > 1 file changed, 5 insertions(+), 1 deletion(-)\n> >> >\n> >> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> >> > index f31a1c8f79..9c0ca5effb 100644\n> >> > --- a/t/test-lib.sh\n> >> > +++ b/t/test-lib.sh\n> >> > @@ -1104,7 +1104,11 @@ test_done () {\n> >> > \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n> >> >\n> >> > \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n> >> > -\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n> >> > +\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n> >> > +\t\t\t\t# try again in a bit\n> >> > +\t\t\t\tsleep 5;\n> >> > +\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n> >> > +\t\t\t} ||\n> >> > \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n> >> > \t\tfi\n> >> > \t\ttest_at_end_hook_\n> >>\n> >> I saw this sleep while reading some test-lib.sh code, doesn't this break\n> >> df4c0d1a79 (test-lib: abort when can't remove trash directory,\n> >> 2017-04-20) for non-Windows platforms?\n> >\n> > It does not really break it, it just delays the inevitable failure.\n\nI still think this is the best answer to this (implicit) question:\n\n> In any case, your patch clearly undoes whatever canary for gc issues\n> df4c0d1a792 was trying to put into the test-lib, but didn't say so in\n> its commit message.\n\nI was not _really_ paying attention to that commit when I implemented the\nwork-around you mentioned above. At the same time I think it does _not_\nundo the canary. If the trash directory cannot be removed via `rm -fr`,\nand if that is an indicator for something fishy going on, chances are that\nthe second `rm -fr` a couple seconds later will _also_ fail, and we still\nget that error message.\n\nThe only reason why the second `rm` should succeed, at least that I can\nthink of, is that something on Windows blocked those files from being\ndeleted, and it is no longer blocking after a couple seconds, and that\nusually means that an anti-malware scanned those files.\n\nCiao,\nDscho\n"},{"id":"420149","messageId":"20210324212051.GA2271@szeder.dev","threadId":"49270","inReplyTo":"nycvar.QRO.7.76.6.2103241257580.50@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v6 16/21] mingw: try to work around issues with the test cleanup","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-03-24T21:20:51Z","receivedAt":"2021-03-24T21:21:41Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Mar 24, 2021 at 01:01:49PM +0100, Johannes Schindelin wrote:\n> Hi Ævar,\n> \n> On Sat, 20 Mar 2021, Ævar Arnfjörð Bjarmason wrote:\n> \n> > On Fri, Mar 19 2021, Johannes Schindelin wrote:\n> >\n> > >> On Tue, Jan 29 2019, Johannes Schindelin via GitGitGadget wrote:\n> > >>\n> > >> > From: Johannes Schindelin \n> > >> >\n> > >> > It seems that every once in a while in the Git for Windows SDK, there\n> > >> > are some transient file locking issues preventing the test clean up to\n> > >> > delete the trash directory. Let's be gentle and try again five seconds\n> > >> > later, and only error out if it still fails the second time.\n> > >> >\n> > >> > This change helps Windows, and does not hurt any other platform\n> > >> > (normally, it is highly unlikely that said deletion fails, and if it\n> > >> > does, normally it will fail again even 5 seconds later).\n> > >> >\n> > >> > Signed-off-by: Johannes Schindelin \n> > >> > ---\n> > >> > t/test-lib.sh | 6 +++++-\n> > >> > 1 file changed, 5 insertions(+), 1 deletion(-)\n> > >> >\n> > >> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> > >> > index f31a1c8f79..9c0ca5effb 100644\n> > >> > --- a/t/test-lib.sh\n> > >> > +++ b/t/test-lib.sh\n> > >> > @@ -1104,7 +1104,11 @@ test_done () {\n> > >> > \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n> > >> >\n> > >> > \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n> > >> > -\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n> > >> > +\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n> > >> > +\t\t\t\t# try again in a bit\n> > >> > +\t\t\t\tsleep 5;\n> > >> > +\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n> > >> > +\t\t\t} ||\n> > >> > \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n> > >> > \t\tfi\n> > >> > \t\ttest_at_end_hook_\n> > >>\n> > >> I saw this sleep while reading some test-lib.sh code, doesn't this break\n> > >> df4c0d1a79 (test-lib: abort when can't remove trash directory,\n> > >> 2017-04-20) for non-Windows platforms?\n> > >\n> > > It does not really break it, it just delays the inevitable failure.\n> \n> I still think this is the best answer to this (implicit) question:\n> \n> > In any case, your patch clearly undoes whatever canary for gc issues\n> > df4c0d1a792 was trying to put into the test-lib, but didn't say so in\n> > its commit message.\n> \n> I was not _really_ paying attention to that commit when I implemented the\n> work-around you mentioned above. At the same time I think it does _not_\n> undo the canary. If the trash directory cannot be removed via `rm -fr`,\n> and if that is an indicator for something fishy going on, chances are that\n> the second `rm -fr` a couple seconds later will _also_ fail, and we still\n> get that error message.\n> \n> The only reason why the second `rm` should succeed, at least that I can\n> think of, is that something on Windows blocked those files from being\n> deleted, and it is no longer blocking after a couple seconds, and that\n> usually means that an anti-malware scanned those files.\n\nBoth commits referenced in df4c0d1a79's log message fixed races\nbetween 'test_done's cleanup and a still running background 'git gc',\nand df4c0d1a79 was meant to draw our attention to similar issues in\nthe future. And it did:\n\n https://public-inbox.org/git/20190602091919.GN951@szeder.dev/\n \nSo no, the failure is not inevitable, there are other reasons why the\nsecond 'rm' might still succeed after the first failed, even just a\nfraction of a second later. And yes, that 'sleep' added in the patch\nabove did unquestionably break that canary,\n\n"},{"id":"420163","messageId":"87lfacp3bp.fsf@evledraar.gmail.com","threadId":"49270","inReplyTo":"20210324212051.GA2271@szeder.dev","subject":"Re: [PATCH v6 16/21] mingw: try to work around issues with the test cleanup","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-03-24T22:57:46Z","receivedAt":"2021-03-24T22:58:43Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Mar 24 2021, SZEDER Gábor wrote:\n\n> On Wed, Mar 24, 2021 at 01:01:49PM +0100, Johannes Schindelin wrote:\n>> Hi Ævar,\n>> \n>> On Sat, 20 Mar 2021, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> > On Fri, Mar 19 2021, Johannes Schindelin wrote:\n>> >\n>> > >> On Tue, Jan 29 2019, Johannes Schindelin via GitGitGadget wrote:\n>> > >>\n>> > >> > From: Johannes Schindelin \n>> > >> >\n>> > >> > It seems that every once in a while in the Git for Windows SDK, there\n>> > >> > are some transient file locking issues preventing the test clean up to\n>> > >> > delete the trash directory. Let's be gentle and try again five seconds\n>> > >> > later, and only error out if it still fails the second time.\n>> > >> >\n>> > >> > This change helps Windows, and does not hurt any other platform\n>> > >> > (normally, it is highly unlikely that said deletion fails, and if it\n>> > >> > does, normally it will fail again even 5 seconds later).\n>> > >> >\n>> > >> > Signed-off-by: Johannes Schindelin \n>> > >> > ---\n>> > >> > t/test-lib.sh | 6 +++++-\n>> > >> > 1 file changed, 5 insertions(+), 1 deletion(-)\n>> > >> >\n>> > >> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n>> > >> > index f31a1c8f79..9c0ca5effb 100644\n>> > >> > --- a/t/test-lib.sh\n>> > >> > +++ b/t/test-lib.sh\n>> > >> > @@ -1104,7 +1104,11 @@ test_done () {\n>> > >> > \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n>> > >> >\n>> > >> > \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n>> > >> > -\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n>> > >> > +\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n>> > >> > +\t\t\t\t# try again in a bit\n>> > >> > +\t\t\t\tsleep 5;\n>> > >> > +\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n>> > >> > +\t\t\t} ||\n>> > >> > \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n>> > >> > \t\tfi\n>> > >> > \t\ttest_at_end_hook_\n>> > >>\n>> > >> I saw this sleep while reading some test-lib.sh code, doesn't this break\n>> > >> df4c0d1a79 (test-lib: abort when can't remove trash directory,\n>> > >> 2017-04-20) for non-Windows platforms?\n>> > >\n>> > > It does not really break it, it just delays the inevitable failure.\n>> \n>> I still think this is the best answer to this (implicit) question:\n>> \n>> > In any case, your patch clearly undoes whatever canary for gc issues\n>> > df4c0d1a792 was trying to put into the test-lib, but didn't say so in\n>> > its commit message.\n>> \n>> I was not _really_ paying attention to that commit when I implemented the\n>> work-around you mentioned above. At the same time I think it does _not_\n>> undo the canary. If the trash directory cannot be removed via `rm -fr`,\n>> and if that is an indicator for something fishy going on, chances are that\n>> the second `rm -fr` a couple seconds later will _also_ fail, and we still\n>> get that error message.\n>> \n>> The only reason why the second `rm` should succeed, at least that I can\n>> think of, is that something on Windows blocked those files from being\n>> deleted, and it is no longer blocking after a couple seconds, and that\n>> usually means that an anti-malware scanned those files.\n>\n> Both commits referenced in df4c0d1a79's log message fixed races\n> between 'test_done's cleanup and a still running background 'git gc',\n> and df4c0d1a79 was meant to draw our attention to similar issues in\n> the future. And it did:\n>\n> https://public-inbox.org/git/20190602091919.GN951@szeder.dev/\n> \n> So no, the failure is not inevitable, there are other reasons why the\n> second 'rm' might still succeed after the first failed, even just a\n> fraction of a second later. And yes, that 'sleep' added in the patch\n> above did unquestionably break that canary,\n\nHaving read that thread now I agree, but I also left with a \"who cares?\"\nand \"so let's keep the sleep then?\".\n\nI.e. is this a problem that any of the software we're maintaining is\ngoing to care about in the wild, it's not like people are expecting gc,\nrepack, fast-import etc. to behave well in the face of rm -rfing the\ndirectory they're operating on.\n\nSo it seems like just an issue that crops up because of how our test\nsuite manages and removes per-test trash directories. So it seems better\nto:\n\n 1. Just keep that \"sleep a bit\" and retry hack\n\n 2. Maybe on some/most platforms we can use cgroups or whatever passes\n for a reliable \"I started a process tree starting at this PID, kill\n -9 the whole thing please\" before cleanup these days.\n"},{"id":"420280","messageId":"20210326195417.GB2271@szeder.dev","threadId":"49270","inReplyTo":"87lfacp3bp.fsf@evledraar.gmail.com","subject":"Re: [PATCH v6 16/21] mingw: try to work around issues with the test cleanup","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2021-03-26T19:54:17Z","receivedAt":"2021-03-26T19:55:26Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Mar 24, 2021 at 11:57:46PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Mar 24 2021, SZEDER Gábor wrote:\n> \n> > On Wed, Mar 24, 2021 at 01:01:49PM +0100, Johannes Schindelin wrote:\n> >> Hi Ævar,\n> >> \n> >> On Sat, 20 Mar 2021, Ævar Arnfjörð Bjarmason wrote:\n> >> \n> >> > On Fri, Mar 19 2021, Johannes Schindelin wrote:\n> >> >\n> >> > >> On Tue, Jan 29 2019, Johannes Schindelin via GitGitGadget wrote:\n> >> > >>\n> >> > >> > From: Johannes Schindelin \n> >> > >> >\n> >> > >> > It seems that every once in a while in the Git for Windows SDK, there\n> >> > >> > are some transient file locking issues preventing the test clean up to\n> >> > >> > delete the trash directory. Let's be gentle and try again five seconds\n> >> > >> > later, and only error out if it still fails the second time.\n> >> > >> >\n> >> > >> > This change helps Windows, and does not hurt any other platform\n> >> > >> > (normally, it is highly unlikely that said deletion fails, and if it\n> >> > >> > does, normally it will fail again even 5 seconds later).\n> >> > >> >\n> >> > >> > Signed-off-by: Johannes Schindelin \n> >> > >> > ---\n> >> > >> > t/test-lib.sh | 6 +++++-\n> >> > >> > 1 file changed, 5 insertions(+), 1 deletion(-)\n> >> > >> >\n> >> > >> > diff --git a/t/test-lib.sh b/t/test-lib.sh\n> >> > >> > index f31a1c8f79..9c0ca5effb 100644\n> >> > >> > --- a/t/test-lib.sh\n> >> > >> > +++ b/t/test-lib.sh\n> >> > >> > @@ -1104,7 +1104,11 @@ test_done () {\n> >> > >> > \t\t\terror \"Tests passed but trash directory already removed before test cleanup; aborting\"\n> >> > >> >\n> >> > >> > \t\t\tcd \"$TRASH_DIRECTORY/..\" &&\n> >> > >> > -\t\t\trm -fr \"$TRASH_DIRECTORY\" ||\n> >> > >> > +\t\t\trm -fr \"$TRASH_DIRECTORY\" || {\n> >> > >> > +\t\t\t\t# try again in a bit\n> >> > >> > +\t\t\t\tsleep 5;\n> >> > >> > +\t\t\t\trm -fr \"$TRASH_DIRECTORY\"\n> >> > >> > +\t\t\t} ||\n> >> > >> > \t\t\terror \"Tests passed but test cleanup failed; aborting\"\n> >> > >> > \t\tfi\n> >> > >> > \t\ttest_at_end_hook_\n> >> > >>\n> >> > >> I saw this sleep while reading some test-lib.sh code, doesn't this break\n> >> > >> df4c0d1a79 (test-lib: abort when can't remove trash directory,\n> >> > >> 2017-04-20) for non-Windows platforms?\n> >> > >\n> >> > > It does not really break it, it just delays the inevitable failure.\n> >> \n> >> I still think this is the best answer to this (implicit) question:\n> >> \n> >> > In any case, your patch clearly undoes whatever canary for gc issues\n> >> > df4c0d1a792 was trying to put into the test-lib, but didn't say so in\n> >> > its commit message.\n> >> \n> >> I was not _really_ paying attention to that commit when I implemented the\n> >> work-around you mentioned above. At the same time I think it does _not_\n> >> undo the canary. If the trash directory cannot be removed via `rm -fr`,\n> >> and if that is an indicator for something fishy going on, chances are that\n> >> the second `rm -fr` a couple seconds later will _also_ fail, and we still\n> >> get that error message.\n> >> \n> >> The only reason why the second `rm` should succeed, at least that I can\n> >> think of, is that something on Windows blocked those files from being\n> >> deleted, and it is no longer blocking after a couple seconds, and that\n> >> usually means that an anti-malware scanned those files.\n> >\n> > Both commits referenced in df4c0d1a79's log message fixed races\n> > between 'test_done's cleanup and a still running background 'git gc',\n> > and df4c0d1a79 was meant to draw our attention to similar issues in\n> > the future. And it did:\n> >\n> > https://public-inbox.org/git/20190602091919.GN951@szeder.dev/\n> > \n> > So no, the failure is not inevitable, there are other reasons why the\n> > second 'rm' might still succeed after the first failed, even just a\n> > fraction of a second later. And yes, that 'sleep' added in the patch\n> > above did unquestionably break that canary,\n> \n> Having read that thread now I agree, but I also left with a \"who cares?\"\n> and \"so let's keep the sleep then?\".\n> \n> I.e. is this a problem that any of the software we're maintaining is\n> going to care about in the wild, it's not like people are expecting gc,\n> repack, fast-import etc. to behave well in the face of rm -rfing the\n> directory they're operating on.\n> \n> So it seems like just an issue that crops up because of how our test\n> suite manages and removes per-test trash directories.\n\nNot at all. The real problem is that some stray background git\nprocess is *still* actively writing to the test repository when the\ntest script is already supposed to be finished.\n\n> So it seems better\n> to:\n> \n> 1. Just keep that \"sleep a bit\" and retry hack\n> \n> 2. Maybe on some/most platforms we can use cgroups or whatever passes\n> for a reliable \"I started a process tree starting at this PID, kill\n> -9 the whole thing please\" before cleanup these days.\n\nWhat really seems better:\n\n 3. Keep that \"sleep a bit\" hack only on platforms that can't delete\n opened files.\n\n"}]}