[QGIS-Developer] Flaky tests and Travis

[Denis Rouzaud]

Another approach would be to have a tag in the PR such as enable_alltests
and it would run on all the tests. I am not sure we have access to labels
in Travis (we didn't have access to title a year ago), but that would be a
nice hack.

On Tue, 12 Mar 2019, 10:42 Alessandro Pasotti, <apasotti at gmail.com> wrote:

> I understand we haven't much of a choice: +1 for disabling.
>
> I wonder if there is a way to make it more evident for developers that
> they need to run disabled tests locally by hand before making a PR.
>
> It happened to me once or twice that I made a change that I knew was
> covered by a test case, let travis run, all green, merge!
>
> And then I discovered that the test I was relying on wasn't running on
> Travis and that there was a test failure when running the test locally.
>
>
>
> On Tue, Mar 12, 2019 at 9:28 AM Matthias Kuhn <matthias at opengis.ch> wrote:
>
>> On 3/12/19 12:28 AM, Nyall Dawson wrote:
>>
>> On Tue, 12 Mar 2019 at 07:59, Denis Rouzaud <denis.rouzaud at gmail.com> <denis.rouzaud at gmail.com> wrote:
>>
>> I am also in favor of disabling them but maybe keep them running and have them as expected failure?
>>
>> That sounds preferable -- but I'm not (personally) sure if it's
>> possible. You know the CI setup better than I do, can you see a way to
>> do this?
>>
>> I'm not sure this works.
>>
>> Expected failure "expects" the tests to fail and will error if it
>> succeeds, since we are dealing with flaky tests here, that will still lead
>> to unstable behavior.
>>
>> If not I'd still be in favor of disabling them.
>>
>> I honestly believe the harm in leaving these enabled (short term) is
>> outweighing the risk of disabling them. We are very likely pushing new
>> contributors away from our project with the current difficulties just
>> getting a PR to green.
>>
>> I agree with the proposal, these tests do more harm than good currently.
>>
>> Matthias
>>
>> Nyall
>>
>>
>>
>> Cheers
>> Denis
>>
>> On Mon, 11 Mar 2019, 22:47 Nyall Dawson, <nyall.dawson at gmail.com> <nyall.dawson at gmail.com> wrote:
>>
>> Hi all,
>>
>> For a long time now we've been plagued by intermittently failing tests
>> on Travis, which are making the whole QGIS development experience
>> quite painful.
>>
>> I propose that we take an absolute hard line approach from now and
>> disable all tests which are causing false positive failures. I've
>> started here: https://github.com/qgis/QGIS/pull/9483
>>
>> This is obviously not ideal, as the failures may be revealing real
>> bugs (and in the case of the two disabled above I believe they are
>> symptoms of the same underlying bug), but I think now we've passed the
>> point where leaving these tests enabled causes more damage then
>> skipping them.
>>
>> Ideally someone would investigate these and fix either the tests or
>> the underlying bugs... but it hasn't happened in 6+ months, so I don't
>> expect that to happen shortly**. I did spend some time around a month
>> ago to see if the fix for these two was trivial, but could not find it
>> quickly.
>>
>> Is anyone opposed to a hard-line "disable if flaky" stance?
>>
>> Nyall
>>
>> ** For full disclosure: next round of QGIS grants I plan on filing for
>> a grant to investigate all tests disabled on Travis in depth and
>> either fix underlying bugs or make the tests more stable. But that's
>> grant dependant, and not a short term solution.
>> _______________________________________________
>> QGIS-Developer mailing listQGIS-Developer at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>> --
>>
>> Denis Rouzauddenis at opengis.ch
>> +41 76 370 21 22
>>
>>
>>
>> _______________________________________________
>> QGIS-Developer mailing listQGIS-Developer at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>> --
>> Matthias Kuhn
>> matthias at opengis.ch
>> +41 (0)76 435 67 63 <+41764356763>
>> [image: OPENGIS.ch Logo] <http://www.opengis.ch>
>> _______________________________________________
>> QGIS-Developer mailing list
>> QGIS-Developer at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>
-- 

Denis Rouzaud
denis at opengis.ch  <denis at opengis.ch>
+41 76 370 21 22
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20190312/4bf946e5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 6671 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20190312/4bf946e5/attachment.png>