[QGIS-Developer] What to do about WFS test failures?

[Alessandro Pasotti]

On Sat, Sep 1, 2018 at 11:56 AM Nyall Dawson <nyall.dawson at gmail.com> wrote:

> Hi devs,
>
> Just raising the question of what we should do about the constant WFS
> test failures we get on Travis. I'd estimate 1 in 3 builds fails
> because of WFS related tests hanging.
>
> I'm very reluctant to disable all these tests, because:
>
> 1. WFS is super important.
> 2. I think there may be a real issue here - I've got at least one
> customer who is having WFS issues with 3.2 and master. BUT: on the
> other hand Travis has always been flaky with any test which uses
> threads, regardless of which area of code it's from. So it could just
> be Travis playing up again, in which case we'd need to disable these
> tests like we do most of the other thread-related tests...
>
> The current situation is basically unworkable. Sooooo.... ideas?
>
> Nyall
>


Sorry I do not have any solution,  the only consideration that comes up to
my mind is that if sum up all the time wasted by developers by
unrelated/random Travis failures we would probably be very badly surprised
(as I usually say Travis is basically a very slow binary entropy generator).

Talking about the tests, WFS tests, and (even for different reasons)
http-based integration tests (some auth tests, qgsfiledownloader, many
server OWS tests) are apparently more fragile than others, mainly because
they depend on (sometimes external) http services, but they have proven in
the past to be very effective in spotting out regressions on very important
feature likes WMS, WFS etc., some of them are also important because they
ensure that QGIS client can talk to its server component.

The very bad things about disabling tests are:

- their development costed a lot of time and efforts and disabling them is
not an incentive for the developers to write more tests [1]
- the tests are obviously important to prevent regressions
- before disabling a test we should make 100% sure that we are not
overlooking a real bug (or what are the tests written for in the first
place?)


So, my recommendation - not very useful in the short term I'm afraid -
would be to start exploring other more reliable options for our CI, even if
they are more expensive.

For the time being, as a temporary solution, there are no other options
than disabling though.

Maybe a good idea would be to allocate some money specifically for the
tests (a dedicated grant, a dedicated hackfest?) just an idea...

[1] of course sometimes it would just mean to write "better" and more
robust tests, but I suspect that this is not generally the case.

-- 
Alessandro Pasotti
w3:   www.itopen.it
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20180901/57a2b8c0/attachment-0001.html>