[QGIS-Developer] Flaky tests and Travis

[Nyall Dawson]

Hi all,

For a long time now we've been plagued by intermittently failing tests
on Travis, which are making the whole QGIS development experience
quite painful.

I propose that we take an absolute hard line approach from now and
disable all tests which are causing false positive failures. I've
started here: https://github.com/qgis/QGIS/pull/9483

This is obviously not ideal, as the failures may be revealing real
bugs (and in the case of the two disabled above I believe they are
symptoms of the same underlying bug), but I think now we've passed the
point where leaving these tests enabled causes more damage then
skipping them.

Ideally someone would investigate these and fix either the tests or
the underlying bugs... but it hasn't happened in 6+ months, so I don't
expect that to happen shortly**. I did spend some time around a month
ago to see if the fix for these two was trivial, but could not find it
quickly.

Is anyone opposed to a hard-line "disable if flaky" stance?

Nyall

** For full disclosure: next round of QGIS grants I plan on filing for
a grant to investigate all tests disabled on Travis in depth and
either fix underlying bugs or make the tests more stable. But that's
grant dependant, and not a short term solution.