[Qgis-psc] Comments from Strk on QGIS-Testsuite

[Sandro Santilli]

On Fri, Feb 19, 2016 at 09:53:29AM +1100, Nyall Dawson wrote:
> On 17 February 2016 at 21:32, Neumann, Andreas <a.neumann at carto.net> wrote:
> >
> > As you may remember, Strk participated in the paid bug fixing program. He
> > made some comments which I am forwarding to the board:
> >
> > --------------------------- Citing -------------------------------
> >
> > This bugfixing drive convinced me even more that QGIS needs to improve its
> > testsuite framework: it was often difficult to figure out how to provide a
> > test for a bug, there was no provision to write a test before fixing a bug
> > (useful to prove the fix is needed), there is only python testing for the
> > whole PostgreSQL provider and it is not automatically setup to run. Shall
> > there be a budget to improve the situation, please let me know as I have a
> > few ideas.
> >
> > ------------------------------------------------------------------
> 
> To be honest, I'm actually of the opposite view. I think QGIS' test
> suite infrastructure is becoming quite mature and reliable.

I would not call this an "opposite view". I didn't say it become worst
or didn't improve (it did, a lot!). Only I think it should keep improving.

> I think the main problem is that there's little to no documentation of
> the test suite. I often see questions posed by strk on the IRC logs
> regarding the tests but I'm never around at the same time to provide
> answers... but there ARE always answers ;)

There's a "Testsuite" category on the bug tracker.
It currently contains 6 open tickets.

Oldest from October 2 2015, newest from February 17 2016.
All mine, apparently :)

Could tickets be a better way to get answers ?
Looking at those 6 tickets, only one received feedback, probably
because I assigned it to Matthias so he got a notification.
Is there any provision, in redmine, to automatically assign tickets
based on category ? (tickets triaging is probably another place where
projet fundings could be directed).

> Having recently added some unit tests to
> GEOS, I'd strongly argue that writing tests for QGIS is much simpler
> ;) (please don't hate me strk!! ;)

No hate, and I agree knowing a path makes it always shorter.

> For python tests you can mark them as expected failures with
> 
> @unittest.expectedFailure (see
> https://docs.python.org/2/library/unittest.html#unittest.expectedFailure)

It's not clear from the documentation if an unexpected success counts
as a failure. That's what I'm after, so that a fix is immediately
secured in a test rather than left as a discarded test.

> For c++ tests I'd "#ifdef 0" until they are fixed.

Same as above, it won't help noticing that a test does exist.

> > there is only python testing for the
> > whole PostgreSQL provider and it is not automatically setup to run.
> 
> I personally don't think this is an issue. Not everyone has access to
> a PostGIS setup for this test to run with, and even if they do they
> may not want running the QGIS test suite to automatically create and
> modify tables on their setup (eg, I would not be happy if the test was
> modifying my production database!!)

The qgis testsuite is using a "qgis_test" database for tests.
Sounds like a safe default. It could be made even safer if you think
it is needed.

Whether or not PostGIS setup is available could be spotted at source
configuration time.

> Again, better test docs would help here.

Surely docs would be an improvement.

> So in summary - I actually think the current test suite setup is good
> (and getting constantly better), and the main problem is both lack of
> testing documentation and training QGIS developers in how to utilise
> it. I'd be happy to run a session during the hackfest explaining all
> this, if that's considered worthwhile...

I'll be there, if available on IRC :)

--strk;