[Qgis-psc] Comments from Strk on QGIS-Testsuite

[Nyall Dawson]

On 22 Feb 2016 9:23 PM, "Sandro Santilli" <strk at keybit.net> wrote:
>
> On Fri, Feb 19, 2016 at 09:53:29AM +1100, Nyall Dawson wrote:
> > On 17 February 2016 at 21:32, Neumann, Andreas <a.neumann at carto.net>
wrote:
> > >
> > > As you may remember, Strk participated in the paid bug fixing
program. He
> > > made some comments which I am forwarding to the board:
> > >
> > > --------------------------- Citing -------------------------------
> > >
> > > This bugfixing drive convinced me even more that QGIS needs to
improve its
> > > testsuite framework: it was often difficult to figure out how to
provide a
> > > test for a bug, there was no provision to write a test before fixing
a bug
> > > (useful to prove the fix is needed), there is only python testing for
the
> > > whole PostgreSQL provider and it is not automatically setup to run.
Shall
> > > there be a budget to improve the situation, please let me know as I
have a
> > > few ideas.
> > >
> > > ------------------------------------------------------------------
> >
> > To be honest, I'm actually of the opposite view. I think QGIS' test
> > suite infrastructure is becoming quite mature and reliable.
>
> I would not call this an "opposite view". I didn't say it become worst
> or didn't improve (it did, a lot!). Only I think it should keep improving.

100% agree! It makes me so happy to see how rapidly the test suite covering
is growing.

>
> > For python tests you can mark them as expected failures with
> >
> > @unittest.expectedFailure (see
> > https://docs.python.org/2/library/unittest.html#unittest.expectedFailure
)
>
> It's not clear from the documentation if an unexpected success counts
> as a failure. That's what I'm after, so that a fix is immediately
> secured in a test rather than left as a discarded test.

It throws a warning, but that's it. It's hard to find proof, but
https://code.djangoproject.com/ticket/23943 seems to suggest that this will
cause a failure in Python 3.0. Not sure.

>
> > For c++ tests I'd "#ifdef 0" until they are fixed.
>
> Same as above, it won't help noticing that a test does exist.

Sorry, bad advice. Looking into this the correct approach is to use the
QEXPECT_FAIL macro:

http://doc.qt.io/qt-4.8/qtest.html#QEXPECT_FAIL

I don't recall seeing anyone put this to use in QGIS yet, but we should do
so where appropriate.

Nyall

>
> > > there is only python testing for the
> > > whole PostgreSQL provider and it is not automatically setup to run.
> >
> > I personally don't think this is an issue. Not everyone has access to
> > a PostGIS setup for this test to run with, and even if they do they
> > may not want running the QGIS test suite to automatically create and
> > modify tables on their setup (eg, I would not be happy if the test was
> > modifying my production database!!)
>
> The qgis testsuite is using a "qgis_test" database for tests.
> Sounds like a safe default. It could be made even safer if you think
> it is needed.
>
> Whether or not PostGIS setup is available could be spotted at source
> configuration time.
>
> > Again, better test docs would help here.
>
> Surely docs would be an improvement.
>
> > So in summary - I actually think the current test suite setup is good
> > (and getting constantly better), and the main problem is both lack of
> > testing documentation and training QGIS developers in how to utilise
> > it. I'd be happy to run a session during the hackfest explaining all
> > this, if that's considered worthwhile...
>
> I'll be there, if available on IRC :)
>
> --strk;
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20160224/b3747759/attachment.html>