[Qgis-psc] Comments from Strk on QGIS-Testsuite

[Matthias Kuhn]

Thanks for this Answer Nyall, my thoughts reading this mail have been
very similar.

Our test suite has improved a lot and its stability is getting better
and better. I am sure that it will help a lot to smooth the transition
to QGIS 3, given that refactoring is very likely to introduce small bugs
which in turn can often be spotted by a good unit test suite.

That does not mean that our test infrastructure is perfect. There is
certainly a lot to improve. The most important part IMO is increasing
the test coverage and getting CI on Windows (and maybe other platforms)
or with different dependency versions. But a perfect test infrastructure
will be a neverending quest for a holy grail anyway :-)

I'll appreciate a broader discussion on what should be financially
supported by the community and the PSC (thanks for starting this
discussion anyway!). And then - if testing is an area to invest -
evaluate which parts of the testing suite needs more attention.

Matthias

On 02/18/2016 11:53 PM, Nyall Dawson wrote:
> On 17 February 2016 at 21:32, Neumann, Andreas <a.neumann at carto.net> wrote:
>> Hi,
>>
>> As you may remember, Strk participated in the paid bug fixing program. He
>> made some comments which I am forwarding to the board:
>>
>> --------------------------- Citing -------------------------------
>>
>> This bugfixing drive convinced me even more that QGIS needs to improve its
>> testsuite framework: it was often difficult to figure out how to provide a
>> test for a bug, there was no provision to write a test before fixing a bug
>> (useful to prove the fix is needed), there is only python testing for the
>> whole PostgreSQL provider and it is not automatically setup to run. Shall
>> there be a budget to improve the situation, please let me know as I have a
>> few ideas.
>>
>> ------------------------------------------------------------------
> To be honest, I'm actually of the opposite view. I think QGIS' test
> suite infrastructure is becoming quite mature and reliable. (aside: In
> fact, more and more test failures are indicative of real bugs rather
> then badly written tests. I'd always discounted the raster layer
> rendering test failures which have been present for years on the
> windows nightly builds to some incorrect paths issue or platform
> specific quirk... When I decided to trust the test suite it revealed
> an extremely nasty bug which has been plaguing Windows users
> http://hub.qgis.org/issues/13155. Anyway, it's fixed now. Wooo for
> actually listening to test failures!).
>
> I think the main problem is that there's little to no documentation of
> the test suite. I often see questions posed by strk on the IRC logs
> regarding the tests but I'm never around at the same time to provide
> answers... but there ARE always answers ;)
>
>> it was often difficult to figure out how to provide a test for a bug
> I think that's always the case until you become familiar with how a
> testing setup is organised. Having recently added some unit tests to
> GEOS, I'd strongly argue that writing tests for QGIS is much simpler
> ;) (please don't hate me strk!! ;)
>
>> there was no provision to write a test before fixing a bug
> For python tests you can mark them as expected failures with
>
> @unittest.expectedFailure (see
> https://docs.python.org/2/library/unittest.html#unittest.expectedFailure)
>
> For c++ tests I'd "#ifdef 0" until they are fixed.
>
>> (useful to prove the fix is needed)
>> there is only python testing for the
>> whole PostgreSQL provider and it is not automatically setup to run.
> I personally don't think this is an issue. Not everyone has access to
> a PostGIS setup for this test to run with, and even if they do they
> may not want running the QGIS test suite to automatically create and
> modify tables on their setup (eg, I would not be happy if the test was
> modifying my production database!!)
>
> Again, better test docs would help here.
>
> So in summary - I actually think the current test suite setup is good
> (and getting constantly better), and the main problem is both lack of
> testing documentation and training QGIS developers in how to utilise
> it. I'd be happy to run a session during the hackfest explaining all
> this, if that's considered worthwhile...
>
> Nyall

-- 
Matthias Kuhn
OPENGIS.ch - https://www.opengis.ch
Spatial • (Q)GIS • PostGIS • Open Source


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20160220/158d6df7/attachment.sig>