[QGIS-Developer] Flaky tests and Travis

[Nyall Dawson]

On Tue, 12 Mar 2019 at 07:59, Denis Rouzaud <denis.rouzaud at gmail.com> wrote:
>
> I am also in favor of disabling them but maybe keep them running and have them as expected failure?

That sounds preferable -- but I'm not (personally) sure if it's
possible. You know the CI setup better than I do, can you see a way to
do this?

> If not I'd still be in favor of disabling them.

I honestly believe the harm in leaving these enabled (short term) is
outweighing the risk of disabling them. We are very likely pushing new
contributors away from our project with the current difficulties just
getting a PR to green.

Nyall


>
> Cheers
> Denis
>
> On Mon, 11 Mar 2019, 22:47 Nyall Dawson, <nyall.dawson at gmail.com> wrote:
>>
>> Hi all,
>>
>> For a long time now we've been plagued by intermittently failing tests
>> on Travis, which are making the whole QGIS development experience
>> quite painful.
>>
>> I propose that we take an absolute hard line approach from now and
>> disable all tests which are causing false positive failures. I've
>> started here: https://github.com/qgis/QGIS/pull/9483
>>
>> This is obviously not ideal, as the failures may be revealing real
>> bugs (and in the case of the two disabled above I believe they are
>> symptoms of the same underlying bug), but I think now we've passed the
>> point where leaving these tests enabled causes more damage then
>> skipping them.
>>
>> Ideally someone would investigate these and fix either the tests or
>> the underlying bugs... but it hasn't happened in 6+ months, so I don't
>> expect that to happen shortly**. I did spend some time around a month
>> ago to see if the fix for these two was trivial, but could not find it
>> quickly.
>>
>> Is anyone opposed to a hard-line "disable if flaky" stance?
>>
>> Nyall
>>
>> ** For full disclosure: next round of QGIS grants I plan on filing for
>> a grant to investigate all tests disabled on Travis in depth and
>> either fix underlying bugs or make the tests more stable. But that's
>> grant dependant, and not a short term solution.
>> _______________________________________________
>> QGIS-Developer mailing list
>> QGIS-Developer at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> --
>
> Denis Rouzaud
> denis at opengis.ch
> +41 76 370 21 22
>
>