Continuous Testing of Git on Windows

Hi team,

at the GitMerge conference, I heard a couple of times "I do not bother reporting bugs in `pu` or `next` on MacOSX and Windows anymore, only Linux seems to be truly supported" or some variation thereof.

This is a strong indicator that we have some room for improvement here.

Active readers of the Git mailing list will not be surprised to read that I think we have to react to build/test failures quicker, that it is not enough to declare it okay for those integration branches to fail the test suite or even to fail to build[*1*].

In that vein, I worked quite a bit on "Continuous Integration", or more appropriately: Continuous Testing. That is, I created an ensemble of jobs that build & test the four integration tests of upstream Git (`maint`, `master`, `next` and `pu`) to verify the *basic* validity of their respective current revisions.

Most CI integrations these days require custom configuration files to be committed, and certain knobs to be twisted on GitHub (which I cannot turn because I have no special privileges on git/git). After struggling with making it work *somehow* anyway (even trying to get in touch with Travis, but they have not bothered to reply to my requests in over a year...), I decided to go with the Visual Studio Team Services (or short, VSTS; it does come in handy that it is developed by distant colleagues of mine, so they *have* to reply to my requests) where the CI configuration can be separated from the source code.

The entire setup is a little bit more complex than your grandfather's CI setup: it has to orchestrate five separate Git repositories, two of them generated and updated from live 32/64-bit Git for Windows SDKs, using a custom pool of build agents due to high resource demands, using three separate Git for Windows installations to support 32/64-bit as well as updating git.exe via `git pull`[*2*].

There is currently only one downside to that setup: the ability to have publicly accessible build logs on VSTS is still in development.

This is not *such* a big downside: if the MacOSX/Linux CI based on Travis[*3*] is any indicator, few people, if any, give a flying, fornicating fly about public build logs.

However, we should strive to improve our software development practices, and one such well-known Best Practice is to use Continuous Testing more effectively, i.e. *not* to ignore it.

That is why I taught the Git for Windows CI job that tests the four upstream Git integration branches to *also* bisect test breakages and then upload comments to the identified commit on GitHub. See an example here (the identified breakage seems to have disappeared in the meantime):

https://github.com/git/git/commit/5a12b3d76973#commitcomment-20802488

The code that generates this comment can be seen here:

https://github.com/git-for-windows/build-extra/blob/50c392c7d107/please.sh#L1648-L1665

So here is hoping to a quicker turnaround from breakage to fix in the future!

Ciao, Johannes

P.S.: I realize that these commit comments may *still* be ignored, but I simply was not yet ready to annoy everybody by having automated mails sent out.

Footnote *1*: It would be kind of okay if, say, `pu` would simply pick up *all* patches so that they would not be forgotten. But that is not the case. Even worse: it was stated recently that the expectation is that the *submitters* of patches find bugs in their code, that the patches should essentially be bug-free by the time they were submitted. This reasoning falls flat on its face considering the very real failures, of course, demonstrating our dear need for Continuous Testing.

Footnote *2*: I will describe the entire setup, including links to the involved repositories, in a separate mail at a later stage.

Footnote *3*: Look at https://travis-ci.org/git/git/builds, and be happy if you have a red/green deficiency so you cannot see all that red.