repo.eclipse.org outage breaking all our linux CI jobs

As https://www.eclipsestatus.io/ reports there is no ETA for recovery, and due to the failure of downloading JGit material in an early stage of our CI jobs, our linux CI jobs are all failing, I am very tempted to apply the following to 'maint' immediately and propagate it all the way up to 'master', 'next', and 'seen'.

I would very very much appreciate additional thoughts and advices by anybody more involved in JGit community and more clueful than I am on the situation.

Anyway, what is somewhat funny is that at the end of this script, there is an attempt to notice and report the lack of jgit (as well as p4 and lfs) but still continuing:

	...
        if type jgit >/dev/null 2>&1
        then
                echo "$(tput setaf 6)JGit Version$(tput sgr0)"
                jgit version
        else
                echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
        fi

        end_group "Install dependencies"

but because ci/lib.sh does "set -e", we fail way before we hit this code. I am tempted to suggest we remove that "set -e" as a long term maintainability improvement measure, but that is a separate topic.

 ci/install-dependencies.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

The ci/install-dependencies.sh script used in a very early phase of our CI jobs downloads Perforce, Git-LFS, and JGit, used for running the test scripts. The test framework is prepared to properly skip the tests that depend on these external software, but the CI script is unnecessarily strict (due to its use of "set -e" in ci/lib.sh) and fails the entire CI run before even starting to test the rest of the system.

Notice a failure to download to any of these external software, but keep going. We need to be careful about cleaning after a failed wget, as a later part of the script that does:

        if type jgit >/dev/null 2>&1
        then
                echo "$(tput setaf 6)JGit Version$(tput sgr0)"
                jgit version
        else
                echo >&2 "WARNING: JGit wasn't installed, see above for clues why"
        fi

will (surprise!) succeed running "type jgit", and then fail with "jgit version", taking the whole thing down due to "set -e".

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 ci/install-dependencies.sh | 31 ++++++++++++++++++++++---------
 1 file changed, 22 insertions(+), 9 deletions(-)

On Thu, Apr 24, 2025 at 04:10:47PM -0700, Junio C Hamano wrote:

Yeah, I think this is a sensible direction to go. It is unfortunate that this may lead to silent breakage of these dependencies unless somebody explicitly looks for those warnings. But that feels like the lesser evil compared to failing the whole pipeline.

I think it would be preferable to only handle failure of wget as chmod shouldn't ever fail if wget was successful. The same is true for the other downloads -- let's be as strict as possible but allow failure of those actions that depend on the network.

Patrick

Patrick Steinhardt <ps@pks.im> writes:

You may think so (as I thought so too before hitting a snag or two) ;-).

The thing is, failing "wget --output-document" leaves an empty file, and an empty file does not still cause "if type jgit; then" to take the "else" clause, and the invocation of "jgit version" would fail, taking the whole thing out, due to the "set -e" thing.

On Fri, Apr 25, 2025 at 02:49:33AM -0700, Junio C Hamano wrote:

But shouldn't the failing wget cause an error, too? So the `|| { }` cleanup branch would execute in that case and we can prune the empty file there. So in other words, shouldn't the following work alright?

    if wget --output-document=...
    then
        massage output
    else
        rm output
    fi

Or am I still missing the obvious?

Patrick

Patrick Steinhardt <ps@pks.im> writes:

While the above "works", what is "obvious" is that it is way too verbose and the merit of going verbose is dubious, especially given that the reason that trigger the argument to favor the above construct over the more concise

	wget && massage || rm

is "massage part should never fail".

We do want to notice a failure in something, if that something is what should never fail, don't we? Not necessarily so!

Our CI jobs are not in the business of checking and ensuring wget or chmod keeps working. If either of them fail, the more important part is its practical impact to the rest of the CI job---resulting "jgit" file is harmful to be left on disk and needs to be removed.

Another to think about this is to imagine if we are having this conversation, had "wget" had (just like its --output-document argument) a "--chmod" argument. (wget && massage) as a unit is conceptually a single "download the jgit binary" in this case, and if either of them fail, we failed to download it.

So, yes, either would "work", but I do not think longhand is warranted in this case.

Hi Patrick,

On Fri, 25 Apr 2025, Patrick Steinhardt wrote:

That is true. It would probably also make sense to mark the message as a `::warning::` on GitHub (and the equivalent on GitLab), so that it is shown a bit more prominently in the CI summary.

Further, as per Matthias Sohn's (i.e. the JGit maintainer's) recommendation at https://discord.com/channels/1042895022950994071/1364872237710184520/1364886912044765216, the JGit download link in particular should probably be changed to https://repo1.maven.org/maven2/org/eclipse/jgit/org.eclipse.jgit.pgm/6.8.0.202311291450-r/org.eclipse.jgit.pgm-6.8.0.202311291450-r.sh (which would work around the CI failures as well and could take the pressure off of working on more graceful dependency management in Git's CI, but then, we don't need more time to discuss the patch in this here thread because it already was fast-tracked to `master`).

Ciao, Johannes

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

::warning:: may be a good idea, but I actually think we should remove the message when we say "ah wget && chmod failed, so let's remove". At the end of the same script, there already is "ah jgit is not there, so let's warn" code exists (and it would falsely say "jgit file exists but cannot be executed" and take the whole thing down if we do not remove after a failed download). The ::warning:: should belong there.

Thanks.

So here is a small update based on Dscho's suggestions to the change we fast-tracked down to 'maint' in order to work around CI failures due to eclipse.org's outage.

Junio C Hamano (2):
  ci: update the message for unavailble third-party software
  ci: download JGit from maven, not eclipse.org

 ci/install-dependencies.sh | 21 ++++++++-------------
 1 file changed, 8 insertions(+), 13 deletions(-)

An earlier fix added an extra message immediately after failing to download a third-party package. But near the end of the script, their availability is checked again and given a message.

Remove the new ones added with a recent fix, as they are redundant. If we were to add more places to download these software (e.g. for other platforms we currently do not download them on), the existing warnning near the end of the script will also trigger.

While at it, as Dscho suggests, rewrite the WARNING: label on the warning message to ::warning::, which presumably should be shown a bit more prominently in the CI summary.

Suggested-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 ci/install-dependencies.sh | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)

As Matthias Sohn, JGit maintainer, recommends, update the JGit download link from repo.eclipse.org to a one in maven.org

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 ci/install-dependencies.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

On Thu, Apr 24, 2025 at 03:13:58PM -0700, Junio C Hamano wrote:

I want to know whether we should use the "cache" mechanism of CI for these third-party softwares? I somehow feel strange that we would download these softwares in every CI.

And if we have the caches, we could somehow avoid problems by hitting the cache when third-party services were down. However, I do not dive into the CI before, so there may be something wrong about my statement.

Thanks, Jialuo

shejialuo <shejialuo@gmail.com> writes:

It also feels wasteful to me that the CI jobs need to do a full install-dependencies.sh over and over, instead of running it once (per platform type), dumping the state, and let all the other jobs on the same platform type to restart from that state ;-).

On Fri, Apr 25, 2025 at 08:20:25AM -0700, Junio C Hamano wrote:

That's right. I'll investigate how to implement this.

On Sat, Apr 26, 2025 at 10:12:51PM +0800, shejialuo wrote:

It would be nice if we could adapt the Linux-based jobs to use pre-seeded Docker images. The idea would be that those images are only built once and then used by later steps of the pipeline. In theory, this could even be extended so that we only rebuild images as-needed when something changes so that the images are reused for multiple pipelines.

Another big benefit would be that this results in a fully-reproducible environment for developers that can in theory be uploaded to a container registry. So if you see that something fails only with a specific job image, you can now trivially fetch that image and try to reproduce the issue in the exact same image as CI used.

Patrick

On Mon, Apr 28, 2025 at 08:49:13AM +0200, Patrick Steinhardt wrote:

I agree that it would be better if we provide pre-built container image. But there is only one problem, which platform we should upload to? I somehow know that github container registry is free for public repository.

So, we may just push the image into github container registry. However, I am not a member of the Git organization. So, I need a help from ones who have the access.

That's right, we could easily set up the development environment to replicate the problem without setting up the environment in our own machine step by step.

Thanks, Jialuo