[Qgis-psc] Handling the Travis CI situation

[Nyall Dawson]

Hi PSC!

There's been a recent thread on the dev list regarding upcoming
changes to the Travis CI service which will impact us greatly. (See
https://lists.osgeo.org/pipermail/qgis-developer/2020-November/062540.html,
https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing ).

In short, while Travis have previously offered us their services for
free they will soon be putting very restrictive limits on the service
for projects like QGIS.

These limits will effectively block us from using Travis. I'd estimate
we average somewhere between 30-40 Travis jobs per day, with each
running somewhere between 30->90 minutes duration.

That said, Travis is a critical part of our project infrastructure. (I
won't argue the importance of CI testing here, that's been done to
death elsewhere and I think we all agree that it's essential for
QGIS).

Denis has started work on moving our CI builds away from Travis to
Github workflows here: https://github.com/qgis/QGIS/pull/39815 . (Full
credit to Denis and big thanks on his prompt response to Travis'
changes!!)

But unfortunately that work has hit a stumbling block in that ~50 of
our ~700 test suites are failing on Github (but pass on Travis). This
isn't an indication that anything is wrong in QGIS, but rather that
the tests are returning slightly different results due to:
- different Qt versions on Github vs Travis
- different underlying library versions (e.g. updated proj, gdal, and
others vs the older Travis build environment)
-  other differences in the build setup, e.g. different limits on the
environment under which the tests run

Before we can make the switch and move away from Travis, we'll need to
update these tests and get as many of them passing as possible on
Github, and then handle the tricky job of backporting the test fixes
and Github action setup to the 3.16/3.10 branches.

This is quite frankly a horrible job to do. It's going to be time
consuming, tedious, and require a LOT of very in depth understanding
of the QGIS code in order to make the correct decisions about whether
a particular difference in test results is revealing a real issue or
just a test which needs updating.

I don't think any one individual will be able to do this, and am
raising the question with PSC of whether qgis.org is able to fund a
coordinated sprint effort for a crack team of devs to resolve this.
(I'd suggest Denis, Alessandro and Matthias would be the ideal
candidates, and I could probably squeeze in up to ~10 hours on this
too).

My estimate is that we're looking at a total commitment of roughly
10-12 developer days effort here. (Including the 3.10/3.16 backports)

Thanks for your consideration!

Nyall