{"thread":{"id":"51299","subject":"[PATCH 1/1] ci: split the `linux-gcc` job into two jobs","startedAt":"2019-06-13T15:20:32Z","lastAt":"2019-06-25T08:55:51Z","messageCount":8,"participants":["Johannes Schindelin via GitGitGadget","SZEDER Gábor","Junio C Hamano","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":1},"messages":[{"id":"377118","messageId":"ac102169ec6f47b8be1fe69b4798de385f102af8.1560430430.git.gitgitgadget@gmail.com","threadId":"51299","inReplyTo":"pull.266.git.gitgitgadget@gmail.com","subject":"[PATCH 1/1] ci: split the `linux-gcc` job into two jobs","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-13T12:53:51Z","receivedAt":"2019-06-13T15:20:32Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nThis job was abused to not only run the test suite in a regular way but\nalso with all kinds of `GIT_TEST_*` options set to non-default values.\n\nLet's split this into two, with the `linux-gcc` job running the default\ntest suite, and the newly-introduced `linux-gcc-extra` job running the\ntest suite in the \"special\" ways.\n\nTechnically, we would have to build Git only once, but it would not be\nobvious how to teach Travis to transport build artifacts, so we keep it\nsimple and just build Git in both jobs.\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n .travis.yml                |  4 ++++\n azure-pipelines.yml        | 39 ++++++++++++++++++++++++++++++++++++++\n ci/install-dependencies.sh |  4 ++--\n ci/lib.sh                  |  4 ++--\n ci/run-build-and-tests.sh  |  5 ++---\n 5 files changed, 49 insertions(+), 7 deletions(-)\n\ndiff --git a/.travis.yml b/.travis.yml\nindex ffb1bc46f2..a6444ee3ab 100644\n--- a/.travis.yml\n+++ b/.travis.yml\n@@ -16,6 +16,10 @@ compiler:\n \n matrix:\n   include:\n+    - env: jobname=linux-gcc-extra\n+      os: linux\n+      compiler: gcc\n+      addons:\n     - env: jobname=GIT_TEST_GETTEXT_POISON\n       os: linux\n       compiler:\ndiff --git a/azure-pipelines.yml b/azure-pipelines.yml\nindex c329b7218b..4080aa3c45 100644\n--- a/azure-pipelines.yml\n+++ b/azure-pipelines.yml\n@@ -206,6 +206,45 @@ jobs:\n       PathtoPublish: t/failed-test-artifacts\n       ArtifactName: failed-test-artifacts\n \n+- job: linux_gcc_extra\n+  displayName: linux-gcc-extra\n+  condition: succeeded()\n+  pool: Hosted Ubuntu 1604\n+  steps:\n+  - bash: |\n+       test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n+\n+       sudo add-apt-repository ppa:ubuntu-toolchain-r/test &&\n+       sudo apt-get update &&\n+       sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2 language-pack-is git-svn gcc-8 || exit 1\n+\n+       export jobname=linux-gcc-extra &&\n+\n+       ci/install-dependencies.sh || exit 1\n+       ci/run-build-and-tests.sh || {\n+           ci/print-test-failures.sh\n+           exit 1\n+       }\n+\n+       test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n+    displayName: 'ci/run-build-and-tests.sh'\n+    env:\n+      GITFILESHAREPWD: $(gitfileshare.pwd)\n+  - task: PublishTestResults@2\n+    displayName: 'Publish Test Results **/TEST-*.xml'\n+    inputs:\n+      mergeTestResults: true\n+      testRunTitle: 'linux-gcc-extra'\n+      platform: Linux\n+      publishRunAttachments: false\n+    condition: succeededOrFailed()\n+  - task: PublishBuildArtifacts@1\n+    displayName: 'Publish trash directories of failed tests'\n+    condition: failed()\n+    inputs:\n+      PathtoPublish: t/failed-test-artifacts\n+      ArtifactName: failed-test-artifacts\n+\n - job: osx_clang\n   displayName: osx-clang\n   condition: succeeded()\ndiff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\nindex 7f6acdd803..c25bdcdf66 100755\n--- a/ci/install-dependencies.sh\n+++ b/ci/install-dependencies.sh\n@@ -9,12 +9,12 @@ P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n+linux-clang|linux-gcc|linux-gcc-extra)\n \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n \tsudo apt-get -q update\n \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2\n \tcase \"$jobname\" in\n-\tlinux-gcc)\n+\tlinux-gcc|linux-gcc-extra)\n \t\tsudo apt-get -q -y install gcc-8\n \t\t;;\n \tesac\ndiff --git a/ci/lib.sh b/ci/lib.sh\nindex 288a5b3884..b16a1454f1 100755\n--- a/ci/lib.sh\n+++ b/ci/lib.sh\n@@ -154,8 +154,8 @@ export DEFAULT_TEST_TARGET=prove\n export GIT_TEST_CLONE_2GB=YesPlease\n \n case \"$jobname\" in\n-linux-clang|linux-gcc)\n-\tif [ \"$jobname\" = linux-gcc ]\n+linux-clang|linux-gcc|linux-gcc-extra)\n+\tif [ \"$jobname\" = linux-gcc -o \"$jobname\" = linux-gcc-extra ]\n \tthen\n \t\texport CC=gcc-8\n \tfi\ndiff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\nindex cdd2913440..b252ff859d 100755\n--- a/ci/run-build-and-tests.sh\n+++ b/ci/run-build-and-tests.sh\n@@ -11,8 +11,7 @@ windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n esac\n \n make\n-make test\n-if test \"$jobname\" = \"linux-gcc\"\n+if test \"$jobname\" = \"linux-gcc-extra\"\n then\n \texport GIT_TEST_SPLIT_INDEX=yes\n \texport GIT_TEST_FULL_IN_PACK_ARRAY=true\n@@ -20,8 +19,8 @@ then\n \texport GIT_TEST_OE_DELTA_SIZE=5\n \texport GIT_TEST_COMMIT_GRAPH=1\n \texport GIT_TEST_MULTI_PACK_INDEX=1\n-\tmake test\n fi\n+make test\n \n check_unignored_build_artifacts\n \n-- \ngitgitgadget\n"},{"id":"377119","messageId":"pull.266.git.gitgitgadget@gmail.com","threadId":"51299","inReplyTo":null,"subject":"[PATCH 0/1] ci: split linux-gcc into linux-gcc and linux-gcc-extra","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-06-13T12:53:50Z","receivedAt":"2019-06-13T15:20:33Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"For people like me, who often look at our CI builds, it is hard to tell\nwhether test suite failures in the linux-gcc job stem from the first make\ntest run, or from the second one, after setting all kinds of GIT_TEST_* \nvariables to non-default values.\n\nLet's make it easier on people like me.\n\nThis also helps the problem where the CI builds often finish the other jobs\nwaaaay before linux-gcc finally finishes, too: linux-gcc and linux-gcc-extra \ncan be run in parallel, on different agents.\n\nJohannes Schindelin (1):\n  ci: split the `linux-gcc` job into two jobs\n\n .travis.yml                |  4 ++++\n azure-pipelines.yml        | 39 ++++++++++++++++++++++++++++++++++++++\n ci/install-dependencies.sh |  4 ++--\n ci/lib.sh                  |  4 ++--\n ci/run-build-and-tests.sh  |  5 ++---\n 5 files changed, 49 insertions(+), 7 deletions(-)\n\n\nbase-commit: b697d92f56511e804b8ba20ccbe7bdc85dc66810\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-266%2Fdscho%2Fsplit-gcc-ci-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-266/dscho/split-gcc-ci-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/266\n-- \ngitgitgadget\n"},{"id":"377126","messageId":"20190613153354.GC31952@szeder.dev","threadId":"51299","inReplyTo":"ac102169ec6f47b8be1fe69b4798de385f102af8.1560430430.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/1] ci: split the `linux-gcc` job into two jobs","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-13T15:33:54Z","receivedAt":"2019-06-13T15:34:02Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jun 13, 2019 at 05:53:51AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> \n> This job was abused to not only run the test suite in a regular way but\n> also with all kinds of `GIT_TEST_*` options set to non-default values.\n> \n> Let's split this into two\n\nWhy...?\n\n> with the `linux-gcc` job running the default\n> test suite, and the newly-introduced `linux-gcc-extra` job running the\n> test suite in the \"special\" ways.\n> \n> Technically, we would have to build Git only once, but it would not be\n> obvious how to teach Travis to transport build artifacts, so we keep it\n> simple and just build Git in both jobs.\n> \n> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n> ---\n>  .travis.yml                |  4 ++++\n>  azure-pipelines.yml        | 39 ++++++++++++++++++++++++++++++++++++++\n>  ci/install-dependencies.sh |  4 ++--\n>  ci/lib.sh                  |  4 ++--\n>  ci/run-build-and-tests.sh  |  5 ++---\n>  5 files changed, 49 insertions(+), 7 deletions(-)\n> \n> diff --git a/.travis.yml b/.travis.yml\n> index ffb1bc46f2..a6444ee3ab 100644\n> --- a/.travis.yml\n> +++ b/.travis.yml\n> @@ -16,6 +16,10 @@ compiler:\n>  \n>  matrix:\n>    include:\n> +    - env: jobname=linux-gcc-extra\n> +      os: linux\n> +      compiler: gcc\n> +      addons:\n>      - env: jobname=GIT_TEST_GETTEXT_POISON\n>        os: linux\n>        compiler:\n> diff --git a/azure-pipelines.yml b/azure-pipelines.yml\n> index c329b7218b..4080aa3c45 100644\n> --- a/azure-pipelines.yml\n> +++ b/azure-pipelines.yml\n> @@ -206,6 +206,45 @@ jobs:\n>        PathtoPublish: t/failed-test-artifacts\n>        ArtifactName: failed-test-artifacts\n>  \n> +- job: linux_gcc_extra\n> +  displayName: linux-gcc-extra\n> +  condition: succeeded()\n> +  pool: Hosted Ubuntu 1604\n> +  steps:\n> +  - bash: |\n> +       test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare \"$GITFILESHAREPWD\" \"$HOME/test-cache\" || exit 1\n> +\n> +       sudo add-apt-repository ppa:ubuntu-toolchain-r/test &&\n> +       sudo apt-get update &&\n> +       sudo apt-get -y install git gcc make libssl-dev libcurl4-openssl-dev libexpat-dev tcl tk gettext git-email zlib1g-dev apache2 language-pack-is git-svn gcc-8 || exit 1\n\nThis installs packages that will be installed by\n'ci/install-dependencies.sh' anyway.\n\n> +\n> +       export jobname=linux-gcc-extra &&\n> +\n> +       ci/install-dependencies.sh || exit 1\n> +       ci/run-build-and-tests.sh || {\n> +           ci/print-test-failures.sh\n> +           exit 1\n> +       }\n> +\n> +       test \"$GITFILESHAREPWD\" = '$(gitfileshare.pwd)' || sudo umount \"$HOME/test-cache\" || exit 1\n> +    displayName: 'ci/run-build-and-tests.sh'\n> +    env:\n> +      GITFILESHAREPWD: $(gitfileshare.pwd)\n> +  - task: PublishTestResults@2\n> +    displayName: 'Publish Test Results **/TEST-*.xml'\n> +    inputs:\n> +      mergeTestResults: true\n> +      testRunTitle: 'linux-gcc-extra'\n> +      platform: Linux\n> +      publishRunAttachments: false\n> +    condition: succeededOrFailed()\n> +  - task: PublishBuildArtifacts@1\n> +    displayName: 'Publish trash directories of failed tests'\n> +    condition: failed()\n> +    inputs:\n> +      PathtoPublish: t/failed-test-artifacts\n> +      ArtifactName: failed-test-artifacts\n> +\n>  - job: osx_clang\n>    displayName: osx-clang\n>    condition: succeeded()\n> diff --git a/ci/install-dependencies.sh b/ci/install-dependencies.sh\n> index 7f6acdd803..c25bdcdf66 100755\n> --- a/ci/install-dependencies.sh\n> +++ b/ci/install-dependencies.sh\n> @@ -9,12 +9,12 @@ P4WHENCE=http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION\n>  LFSWHENCE=https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION\n>  \n>  case \"$jobname\" in\n> -linux-clang|linux-gcc)\n> +linux-clang|linux-gcc|linux-gcc-extra)\n>  \tsudo apt-add-repository -y \"ppa:ubuntu-toolchain-r/test\"\n>  \tsudo apt-get -q update\n>  \tsudo apt-get -q -y install language-pack-is libsvn-perl apache2\n>  \tcase \"$jobname\" in\n> -\tlinux-gcc)\n> +\tlinux-gcc|linux-gcc-extra)\n>  \t\tsudo apt-get -q -y install gcc-8\n>  \t\t;;\n>  \tesac\n> diff --git a/ci/lib.sh b/ci/lib.sh\n> index 288a5b3884..b16a1454f1 100755\n> --- a/ci/lib.sh\n> +++ b/ci/lib.sh\n> @@ -154,8 +154,8 @@ export DEFAULT_TEST_TARGET=prove\n>  export GIT_TEST_CLONE_2GB=YesPlease\n>  \n>  case \"$jobname\" in\n> -linux-clang|linux-gcc)\n> -\tif [ \"$jobname\" = linux-gcc ]\n> +linux-clang|linux-gcc|linux-gcc-extra)\n> +\tif [ \"$jobname\" = linux-gcc -o \"$jobname\" = linux-gcc-extra ]\n>  \tthen\n>  \t\texport CC=gcc-8\n>  \tfi\n> diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh\n> index cdd2913440..b252ff859d 100755\n> --- a/ci/run-build-and-tests.sh\n> +++ b/ci/run-build-and-tests.sh\n> @@ -11,8 +11,7 @@ windows*) cmd //c mklink //j t\\\\.prove \"$(cygpath -aw \"$cache_dir/.prove\")\";;\n>  esac\n>  \n>  make\n> -make test\n> -if test \"$jobname\" = \"linux-gcc\"\n> +if test \"$jobname\" = \"linux-gcc-extra\"\n>  then\n>  \texport GIT_TEST_SPLIT_INDEX=yes\n>  \texport GIT_TEST_FULL_IN_PACK_ARRAY=true\n> @@ -20,8 +19,8 @@ then\n>  \texport GIT_TEST_OE_DELTA_SIZE=5\n>  \texport GIT_TEST_COMMIT_GRAPH=1\n>  \texport GIT_TEST_MULTI_PACK_INDEX=1\n> -\tmake test\n>  fi\n> +make test\n>  \n>  check_unignored_build_artifacts\n>  \n> -- \n> gitgitgadget\n"},{"id":"377128","messageId":"xmqqv9x9wkba.fsf@gitster-ct.c.googlers.com","threadId":"51299","inReplyTo":"20190613153354.GC31952@szeder.dev","subject":"Re: [PATCH 1/1] ci: split the `linux-gcc` job into two jobs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-06-13T15:56:25Z","receivedAt":"2019-06-13T15:56:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor <szeder.dev@gmail.com> writes:\n\n> On Thu, Jun 13, 2019 at 05:53:51AM -0700, Johannes Schindelin via GitGitGadget wrote:\n>> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n>> \n>> This job was abused to not only run the test suite in a regular way but\n>> also with all kinds of `GIT_TEST_*` options set to non-default values.\n>> \n>> Let's split this into two\n>\n> Why...?\n>\n>> with the `linux-gcc` job running the default\n>> test suite, and the newly-introduced `linux-gcc-extra` job running the\n>> test suite in the \"special\" ways.\n>> \n>> Technically, we would have to build Git only once, but it would not be\n>> obvious how to teach Travis to transport build artifacts, so we keep it\n>> simple and just build Git in both jobs.\n\nI had the same reaction.\n\nIf it said something like:\n\n    There is no logical reason why these extras need to be tied to\n    the linux-gcc platform.  instead of tying the extra\n    configuration tests only to linux-gcc, split it so that they are\n    also run on all other combinations, and this is merely a first\n    step of doing so\n\nit might at least have been possible to judge if the motivation is\nsane, but the proposed log message, while it is quite clear what is\nbeing done and what its shortcomings are, is silent on why we would\nwant to do this in the first place, and that makes it even harder to\nswallow the shortcomings.\n\nThanks.\n"},{"id":"377141","messageId":"nycvar.QRO.7.76.6.1906131844440.44@tvgsbejvaqbjf.bet","threadId":"51299","inReplyTo":"xmqqv9x9wkba.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH 1/1] ci: split the `linux-gcc` job into two jobs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-13T16:51:04Z","receivedAt":"2019-06-13T16:51:05Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 13 Jun 2019, Junio C Hamano wrote:\n\n> SZEDER Gábor <szeder.dev@gmail.com> writes:\n>\n> > On Thu, Jun 13, 2019 at 05:53:51AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> >> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> >>\n> >> This job was abused to not only run the test suite in a regular way but\n> >> also with all kinds of `GIT_TEST_*` options set to non-default values.\n> >>\n> >> Let's split this into two\n> >\n> > Why...?\n> >\n> >> with the `linux-gcc` job running the default\n> >> test suite, and the newly-introduced `linux-gcc-extra` job running the\n> >> test suite in the \"special\" ways.\n> >>\n> >> Technically, we would have to build Git only once, but it would not be\n> >> obvious how to teach Travis to transport build artifacts, so we keep it\n> >> simple and just build Git in both jobs.\n>\n> I had the same reaction.\n\nSo basically you are saying that the cover letter was the wrong location\nfor this:\n\n\tFor people like me, who often look at our CI builds, it is hard to\n\ttell whether test suite failures in the linux-gcc job stem from\n\tthe first make test run, or from the second one, after setting all\n\tkinds of GIT_TEST_* variables to non-default values.\n\n\tLet's make it easier on people like me.\n\n\tThis also helps the problem where the CI builds often finish the other\n\tjobs waaaay before linux-gcc finally finishes, too: linux-gcc and\n\tlinux-gcc-extra can be run in parallel, on different agents.\n\nOf course, I would rephrase that a little, because it is kinda okay for a\ncover letter, but not for a commit message that needs to be understandable\nout of context.\n\nIf you agree that this would make the change easier to swallow, I will\nmake it so.\n\n> If it said something like:\n>\n>     There is no logical reason why these extras need to be tied to\n>     the linux-gcc platform.  instead of tying the extra\n>     configuration tests only to linux-gcc, split it so that they are\n>     also run on all other combinations, and this is merely a first\n>     step of doing so\n\nNo, that would double the time taken by the CI build. It already taxes the\npatience so much that most contributors already don't bother with it, and\nyou can see how many times I find issues and have to report them (which\nflies in the face of the idea of using automated builds to catch bugs\nearly): we are seeing the detrimental effects of a regression test suite\nthat takes too long.\n\n> it might at least have been possible to judge if the motivation is\n> sane, but the proposed log message, while it is quite clear what is\n> being done and what its shortcomings are, is silent on why we would\n> want to do this in the first place, and that makes it even harder to\n> swallow the shortcomings.\n\nOkay, two things:\n\n\t- it makes it easier to identify *which* settings to use when a\n\t  test case fails in `linux-gcc` or `linux-gcc-extra`.\n\n\t- the overall CI build runtime is no longer dictated by the\n\t  runtime of this single build, which easily was the slowest of\n\t  the bunch.\n\nDoes that clear up the picture?\nDscho\n"},{"id":"377160","messageId":"20190613174329.GE31952@szeder.dev","threadId":"51299","inReplyTo":"nycvar.QRO.7.76.6.1906131844440.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/1] ci: split the `linux-gcc` job into two jobs","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2019-06-13T17:43:29Z","receivedAt":"2019-06-13T17:43:35Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jun 13, 2019 at 06:51:04PM +0200, Johannes Schindelin wrote:\n> Hi,\n> \n> On Thu, 13 Jun 2019, Junio C Hamano wrote:\n> \n> > SZEDER Gábor <szeder.dev@gmail.com> writes:\n> >\n> > > On Thu, Jun 13, 2019 at 05:53:51AM -0700, Johannes Schindelin via GitGitGadget wrote:\n> > >> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> > >>\n> > >> This job was abused to not only run the test suite in a regular way but\n> > >> also with all kinds of `GIT_TEST_*` options set to non-default values.\n> > >>\n> > >> Let's split this into two\n> > >\n> > > Why...?\n> > >\n> > >> with the `linux-gcc` job running the default\n> > >> test suite, and the newly-introduced `linux-gcc-extra` job running the\n> > >> test suite in the \"special\" ways.\n> > >>\n> > >> Technically, we would have to build Git only once, but it would not be\n> > >> obvious how to teach Travis to transport build artifacts, so we keep it\n> > >> simple and just build Git in both jobs.\n> >\n> > I had the same reaction.\n> \n> So basically you are saying that the cover letter was the wrong location\n> for this:\n> \n> \tFor people like me, who often look at our CI builds, it is hard to\n> \ttell whether test suite failures in the linux-gcc job stem from\n> \tthe first make test run, or from the second one, after setting all\n> \tkinds of GIT_TEST_* variables to non-default values.\n\nIs this really an issue in practice?  In my experience there are only\ntwo (and a half) cases:\n\n  - if both the 'linux-gcc' and 'linux-clang' build jobs fail, then\n    it's some sort of a general breakage.\n\n  - if only the 'linux-gcc' build job fails, the 'linux-clang'\n    succeeds, then it's a breakage in the test run with the various\n    'GIT_TEST_*' test knobs enabled (unless the failing 'linux-gcc'\n    build job's runtime is below, say, 5 minutes, in which case it's a\n    build error only triggered by GCC(-8), and, as I recall, is rather\n    rare).\n\n\n> \tLet's make it easier on people like me.\n> \n> \tThis also helps the problem where the CI builds often finish the other\n> \tjobs waaaay before linux-gcc finally finishes\n\nThis is not the case on Travis CI, where the runtime of the macOS\nbuild jobs are far the longest, so this change won't help anything\nthere... on the contrary, it would make things slower by spending time\non installing dependencies and building Git in one more build job.\n\n>       too: linux-gcc and\n> \tlinux-gcc-extra can be run in parallel, on different agents.\n"},{"id":"377252","messageId":"nycvar.QRO.7.76.6.1906142051300.44@tvgsbejvaqbjf.bet","threadId":"51299","inReplyTo":"20190613174329.GE31952@szeder.dev","subject":"Re: [PATCH 1/1] ci: split the `linux-gcc` job into two jobs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-14T19:35:26Z","receivedAt":"2019-06-14T19:35:25Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Gábor,\n\nOn Thu, 13 Jun 2019, SZEDER Gábor wrote:\n\n> On Thu, Jun 13, 2019 at 06:51:04PM +0200, Johannes Schindelin wrote:\n>\n> > On Thu, 13 Jun 2019, Junio C Hamano wrote:\n> >\n> > > SZEDER Gábor <szeder.dev@gmail.com> writes:\n> > >\n> > > > On Thu, Jun 13, 2019 at 05:53:51AM -0700, Johannes Schindelin via\n> > > > GitGitGadget wrote:\n> > > >> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> > > >>\n> > > >> This job was abused to not only run the test suite in a regular way but\n> > > >> also with all kinds of `GIT_TEST_*` options set to non-default values.\n> > > >>\n> > > >> Let's split this into two\n> > > >\n> > > > Why...?\n> > > >\n> > > >> with the `linux-gcc` job running the default\n> > > >> test suite, and the newly-introduced `linux-gcc-extra` job running the\n> > > >> test suite in the \"special\" ways.\n> > > >>\n> > > >> Technically, we would have to build Git only once, but it would not be\n> > > >> obvious how to teach Travis to transport build artifacts, so we keep it\n> > > >> simple and just build Git in both jobs.\n> > >\n> > > I had the same reaction.\n> >\n> > So basically you are saying that the cover letter was the wrong location\n> > for this:\n> >\n> > \tFor people like me, who often look at our CI builds, it is hard to\n> > \ttell whether test suite failures in the linux-gcc job stem from\n> > \tthe first make test run, or from the second one, after setting all\n> > \tkinds of GIT_TEST_* variables to non-default values.\n>\n> Is this really an issue in practice?\n\nI don't think that this is the right question. The right question would\nbe: is this issue possible? And the answer is: yes, quite. The clang and\nthe GCC toolchains are different enough that they have different bugs and\nstrengths. And the test suite with extra knobs vs without them *also* is\ndifferent enough to expose different bugs. So obviously, you would want\nto discern between them [*1*].\n\nBut I can even answer the wrong question. The answer is still: yes, quite.\n\nFor example, I saw quite a few flaky tests \"prefer\" one over the other. I\ndo not recall the specifics (as I investigated at least half a dozeny\nflaky tests in the past months, and I am prone to confuse them with one\nanother), but I distinctly remember debugging via patching\nazure-pipelines.yml and ci/ heavily, using the one job that was failing *a\nlot* more often (and deleting all the other jobs from that .yml file,\nwhich accelerated the turn-around time, which is *everything* in\ndebugging).\n\nAnd even if I had not experienced this. As I said, clang and GCC are\ndifferent enough, that's why we have both jobs in the first place. It\nsounds rather curious to me that you suggest that they essentially do the\nsame further below:\n\n> In my experience there are only two (and a half) cases:\n>\n>   - if both the 'linux-gcc' and 'linux-clang' build jobs fail, then\n>     it's some sort of a general breakage.\n\nSure, that's the easy case.\n\nWhat I want to help with this patch is the *hard* cases.\n\n>   - if only the 'linux-gcc' build job fails, the 'linux-clang'\n>     succeeds, then it's a breakage in the test run with the various\n>     'GIT_TEST_*' test knobs enabled (unless the failing 'linux-gcc'\n>     build job's runtime is below, say, 5 minutes, in which case it's a\n>     build error only triggered by GCC(-8), and, as I recall, is rather\n>     rare).\n\nSo what you are suggesting is that the part of the `linux-gcc` job where\nit tests without all those knobs is totally useless because `linux-clang`\nalready tested the same stuff?\n\nThat does not sound right.\n\nBecause by that token, you would want to simply remove that part from the\n`linux-gcc` job (instead of splitting out the rest, as my patch does).\n\nI refuse to believe that you are syaing that.\n\nThat would sound almost like \"We don't need the test suite because 99.9%\nof all test cases pass, anyway\".\n\n> > \tLet's make it easier on people like me.\n> >\n> > \tThis also helps the problem where the CI builds often finish the\n> > \tother jobs waaaay before linux-gcc finally finishes\n>\n> This is not the case on Travis CI, where the runtime of the macOS\n> build jobs are far the longest, so this change won't help anything\n> there...\n\nRight, Travis' macOS agents are ridiculously slow.\n\n> on the contrary, it would make things slower by spending time on\n> installing dependencies and building Git in one more build job.\n\nNo, it wouldn't. Because instead of waiting for the macOS jobs and the\nlinux-gcc job, we would only wait for the macOS jobs.\n\nThe fallacy here is that the 2-3 minutes spent in *two* instead of *one*\nagent would accumulate to 2-3 minutes. It's parallel instead.\n\nAnd once Travis gets faster macOS agents, the Travis build will be overall\nfaster (instead of now waiting for `linux-gcc` all the time).\n\nOr am I missing anything obvious? I am quite puzzled by your objections,\ngiven your experience with the CI builds. You, too, have *got* to have\nexperienced the benefits of parallelizing longer-running jobs.\n\nTo me, it looks like a no-brainer to split apart a long-running job, to\nbenefit from running jobs side by side.\n\nOf course, there is also the presentation of the test results, but then,\nTravis does not have that. You cannot publish the test results in a visual\nmanner, nor analyze breakages over time. So in Travis, it does matter less\nthan in Azure Pipelines (although not by much) what is the name of the job\nin which a test failed, it really leaves the developer struggling to get\nto the root cause by digging through the entire log. In Azure Pipelines, I\nclick on the Tests tab (see e.g.\nhttps://dev.azure.com/git/git/_build/results?buildId=677&view=ms.vss-test-web.build-test-results-tab)\nand I see immediately not only what test script, not only what test case\nfailed, being able to see the corresponding part of the verbose output by\nclicking on the test case title, I also immediately see in what job it\nfailed, which can help me debug a lot faster. Also, the analytics section\nallows me to see in which jobs tests failed consistently.\n\nAnd with the split I proposed, it would be obvious from that page, at one\nglance, whether I need to use the GIT_TEST_* knobs to reproduce a test\nfailure locally or not.\n\nSo: I am still very, very puzzled why you think it to be a good idea to\nhave a job that runs twice as long as all the other Linux jobs, that makes\nregressions harder to investigate than necessary, and that makes the\noverall analysis e.g. of flaky tests more difficult than with my patch.\n\nCiao,\nDscho\n\nFootnote *1*: Now, a question that Junio raised was whether we should have\nthe test runs with the GIT_TEST_* knobs *also* for clang. Alas, here I\nwould like to throw in the argument that a \"too complete\" test suite is so\nuseless as to be a wasted effort because *nobody runs it if it takes too\nlong*. And given the impression I have that Junio does not bother looking\nat the CI builds, I wonder why he wanted this in the first place, it's not\nlike it would benefit him.\n\n> >       too: linux-gcc and\n> > \tlinux-gcc-extra can be run in parallel, on different agents.\n>\n"},{"id":"377944","messageId":"nycvar.QRO.7.76.6.1906251053160.44@tvgsbejvaqbjf.bet","threadId":"51299","inReplyTo":"nycvar.QRO.7.76.6.1906142051300.44@tvgsbejvaqbjf.bet","subject":"Re: [PATCH 1/1] ci: split the `linux-gcc` job into two jobs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-25T08:56:06Z","receivedAt":"2019-06-25T08:55:51Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Gábor,\n\nOn Fri, 14 Jun 2019, Johannes Schindelin wrote:\n\n> On Thu, 13 Jun 2019, SZEDER Gábor wrote:\n>\n> > On Thu, Jun 13, 2019 at 06:51:04PM +0200, Johannes Schindelin wrote:\n> >\n> > > On Thu, 13 Jun 2019, Junio C Hamano wrote:\n> > >\n> > > > SZEDER Gábor <szeder.dev@gmail.com> writes:\n> > > >\n> > > > > On Thu, Jun 13, 2019 at 05:53:51AM -0700, Johannes Schindelin via\n> > > > > GitGitGadget wrote:\n> > > > >> From: Johannes Schindelin <johannes.schindelin@gmx.de>\n> > > > >>\n> > > > >> This job was abused to not only run the test suite in a regular\n> > > > >> way but also with all kinds of `GIT_TEST_*` options set to\n> > > > >> non-default values.\n> > > > >>\n> > > > >> Let's split this into two\n> > > > >\n> > > > > Why...?\n> > > > >\n> > > > >> with the `linux-gcc` job running the default test suite, and\n> > > > >> the newly-introduced `linux-gcc-extra` job running the test\n> > > > >> suite in the \"special\" ways.\n> > > > >>\n> > > > >> Technically, we would have to build Git only once, but it would\n> > > > >> not be obvious how to teach Travis to transport build\n> > > > >> artifacts, so we keep it simple and just build Git in both\n> > > > >> jobs.\n> > > >\n> > > > I had the same reaction.\n> > >\n> > > So basically you are saying that the cover letter was the wrong\n> > > location for this:\n> > >\n> > > \tFor people like me, who often look at our CI builds, it is hard to\n> > > \ttell whether test suite failures in the linux-gcc job stem from\n> > > \tthe first make test run, or from the second one, after setting all\n> > > \tkinds of GIT_TEST_* variables to non-default values.\n> >\n> > Is this really an issue in practice?\n>\n> I don't think that this is the right question.\n\nI still think that this is the wrong question.\n\nTo put more water down the drain, I would like to challenge you to look at\nthis build and tell me as fast as you can what half of the linux-gcc job\nfails, and whether the other half of the job fails, too, or whether the\ntest cases succeed there, and if so, why:\n\nhttps://dev.azure.com/gitgitgadget/git/_build/results?buildId=11410&view=ms.vss-test-web.build-test-results-tab\n\nWe really need to split linux-gcc. It's not right that it throws two\ncompletely separate concerns into the same bucket.\n\nCiao,\nDscho\n"}]}