[QGIS-Developer] Flaky tests and Travis

[Matthias Kuhn]

On 3/12/19 12:28 AM, Nyall Dawson wrote:
> On Tue, 12 Mar 2019 at 07:59, Denis Rouzaud <denis.rouzaud at gmail.com> wrote:
>> I am also in favor of disabling them but maybe keep them running and have them as expected failure?
> That sounds preferable -- but I'm not (personally) sure if it's
> possible. You know the CI setup better than I do, can you see a way to
> do this?

I'm not sure this works.

Expected failure "expects" the tests to fail and will error if it
succeeds, since we are dealing with flaky tests here, that will still
lead to unstable behavior.

>
>> If not I'd still be in favor of disabling them.
> I honestly believe the harm in leaving these enabled (short term) is
> outweighing the risk of disabling them. We are very likely pushing new
> contributors away from our project with the current difficulties just
> getting a PR to green.

I agree with the proposal, these tests do more harm than good currently.

Matthias

>
> Nyall
>
>
>> Cheers
>> Denis
>>
>> On Mon, 11 Mar 2019, 22:47 Nyall Dawson, <nyall.dawson at gmail.com> wrote:
>>> Hi all,
>>>
>>> For a long time now we've been plagued by intermittently failing tests
>>> on Travis, which are making the whole QGIS development experience
>>> quite painful.
>>>
>>> I propose that we take an absolute hard line approach from now and
>>> disable all tests which are causing false positive failures. I've
>>> started here: https://github.com/qgis/QGIS/pull/9483
>>>
>>> This is obviously not ideal, as the failures may be revealing real
>>> bugs (and in the case of the two disabled above I believe they are
>>> symptoms of the same underlying bug), but I think now we've passed the
>>> point where leaving these tests enabled causes more damage then
>>> skipping them.
>>>
>>> Ideally someone would investigate these and fix either the tests or
>>> the underlying bugs... but it hasn't happened in 6+ months, so I don't
>>> expect that to happen shortly**. I did spend some time around a month
>>> ago to see if the fix for these two was trivial, but could not find it
>>> quickly.
>>>
>>> Is anyone opposed to a hard-line "disable if flaky" stance?
>>>
>>> Nyall
>>>
>>> ** For full disclosure: next round of QGIS grants I plan on filing for
>>> a grant to investigate all tests disabled on Travis in depth and
>>> either fix underlying bugs or make the tests more stable. But that's
>>> grant dependant, and not a short term solution.
>>> _______________________________________________
>>> QGIS-Developer mailing list
>>> QGIS-Developer at lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> --
>>
>> Denis Rouzaud
>> denis at opengis.ch
>> +41 76 370 21 22
>>
>>
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
-- 
Matthias Kuhn
matthias at opengis.ch <mailto:matthias at opengis.ch>
+41 (0)76 435 67 63 <tel:+41764356763>
OPENGIS.ch Logo <http://www.opengis.ch>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20190312/6c708de5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 6671 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20190312/6c708de5/attachment.png>