[Qgis-psc] Comments from Strk on QGIS-Testsuite

[Nyall Dawson]

On 17 February 2016 at 21:32, Neumann, Andreas <a.neumann at carto.net> wrote:
> Hi,
>
> As you may remember, Strk participated in the paid bug fixing program. He
> made some comments which I am forwarding to the board:
>
> --------------------------- Citing -------------------------------
>
> This bugfixing drive convinced me even more that QGIS needs to improve its
> testsuite framework: it was often difficult to figure out how to provide a
> test for a bug, there was no provision to write a test before fixing a bug
> (useful to prove the fix is needed), there is only python testing for the
> whole PostgreSQL provider and it is not automatically setup to run. Shall
> there be a budget to improve the situation, please let me know as I have a
> few ideas.
>
> ------------------------------------------------------------------

To be honest, I'm actually of the opposite view. I think QGIS' test
suite infrastructure is becoming quite mature and reliable. (aside: In
fact, more and more test failures are indicative of real bugs rather
then badly written tests. I'd always discounted the raster layer
rendering test failures which have been present for years on the
windows nightly builds to some incorrect paths issue or platform
specific quirk... When I decided to trust the test suite it revealed
an extremely nasty bug which has been plaguing Windows users
http://hub.qgis.org/issues/13155. Anyway, it's fixed now. Wooo for
actually listening to test failures!).

I think the main problem is that there's little to no documentation of
the test suite. I often see questions posed by strk on the IRC logs
regarding the tests but I'm never around at the same time to provide
answers... but there ARE always answers ;)

> it was often difficult to figure out how to provide a test for a bug

I think that's always the case until you become familiar with how a
testing setup is organised. Having recently added some unit tests to
GEOS, I'd strongly argue that writing tests for QGIS is much simpler
;) (please don't hate me strk!! ;)

> there was no provision to write a test before fixing a bug

For python tests you can mark them as expected failures with

@unittest.expectedFailure (see
https://docs.python.org/2/library/unittest.html#unittest.expectedFailure)

For c++ tests I'd "#ifdef 0" until they are fixed.

> (useful to prove the fix is needed)

> there is only python testing for the
> whole PostgreSQL provider and it is not automatically setup to run.

I personally don't think this is an issue. Not everyone has access to
a PostGIS setup for this test to run with, and even if they do they
may not want running the QGIS test suite to automatically create and
modify tables on their setup (eg, I would not be happy if the test was
modifying my production database!!)

Again, better test docs would help here.

So in summary - I actually think the current test suite setup is good
(and getting constantly better), and the main problem is both lack of
testing documentation and training QGIS developers in how to utilise
it. I'd be happy to run a session during the hackfest explaining all
this, if that's considered worthwhile...

Nyall