[Qgis-psc] Comments from Strk on QGIS-Testsuite

Sat Feb 20 09:40:21 PST 2016

Hi

> On 20 Feb 2016, at 16:38, Matthias Kuhn <matthias at opengis.ch> wrote:
> 
> Thanks for this Answer Nyall, my thoughts reading this mail have been
> very similar.
> 
> Our test suite has improved a lot and its stability is getting better
> and better. I am sure that it will help a lot to smooth the transition
> to QGIS 3, given that refactoring is very likely to introduce small bugs
> which in turn can often be spotted by a good unit test suite.
> 
> That does not mean that our test infrastructure is perfect. There is
> certainly a lot to improve. The most important part IMO is increasing
> the test coverage and getting CI on Windows (and maybe other platforms)
> or with different dependency versions. But a perfect test infrastructure
> will be a neverending quest for a holy grail anyway :-)
> 
> I'll appreciate a broader discussion on what should be financially
> supported by the community and the PSC (thanks for starting this
> discussion anyway!). And then - if testing is an area to invest -
> evaluate which parts of the testing suite needs more attention.

Having the CI stuff going has been a huge leap forward and with my QA PSC member hat on, I would say it is definitely something to invest more time and effort into. As I mentioned before though I would just like us to be more strategic about our investments (from the PSC side anyway) and ensure that we put more focus on other places that urgently need support (like getting decent API docs in place to name a random example). Thanks for all the great stuff you have done with the test suite Matthias, looking forward to getting more stuff done on the testing front in the future. I also think we need to start thinking about GUI testing soon - GUI testing will put a lot of the underlying API’s to task and also ensure that fronted behaviour is consistent from release to release (or at least that it is consciously changed instead of unconsciously).

Regards

Tim

> 
> Matthias
> 
> On 02/18/2016 11:53 PM, Nyall Dawson wrote:
>> On 17 February 2016 at 21:32, Neumann, Andreas <a.neumann at carto.net> wrote:
>>> Hi,
>>> 
>>> As you may remember, Strk participated in the paid bug fixing program. He
>>> made some comments which I am forwarding to the board:
>>> 
>>> --------------------------- Citing -------------------------------
>>> 
>>> This bugfixing drive convinced me even more that QGIS needs to improve its
>>> testsuite framework: it was often difficult to figure out how to provide a
>>> test for a bug, there was no provision to write a test before fixing a bug
>>> (useful to prove the fix is needed), there is only python testing for the
>>> whole PostgreSQL provider and it is not automatically setup to run. Shall
>>> there be a budget to improve the situation, please let me know as I have a
>>> few ideas.
>>> 
>>> ------------------------------------------------------------------
>> To be honest, I'm actually of the opposite view. I think QGIS' test
>> suite infrastructure is becoming quite mature and reliable. (aside: In
>> fact, more and more test failures are indicative of real bugs rather
>> then badly written tests. I'd always discounted the raster layer
>> rendering test failures which have been present for years on the
>> windows nightly builds to some incorrect paths issue or platform
>> specific quirk... When I decided to trust the test suite it revealed
>> an extremely nasty bug which has been plaguing Windows users
>> http://hub.qgis.org/issues/13155. Anyway, it's fixed now. Wooo for
>> actually listening to test failures!).
>> 
>> I think the main problem is that there's little to no documentation of
>> the test suite. I often see questions posed by strk on the IRC logs
>> regarding the tests but I'm never around at the same time to provide
>> answers... but there ARE always answers ;)
>> 
>>> it was often difficult to figure out how to provide a test for a bug
>> I think that's always the case until you become familiar with how a
>> testing setup is organised. Having recently added some unit tests to
>> GEOS, I'd strongly argue that writing tests for QGIS is much simpler
>> ;) (please don't hate me strk!! ;)
>> 
>>> there was no provision to write a test before fixing a bug
>> For python tests you can mark them as expected failures with
>> 
>> @unittest.expectedFailure (see
>> https://docs.python.org/2/library/unittest.html#unittest.expectedFailure)
>> 
>> For c++ tests I'd "#ifdef 0" until they are fixed.
>> 
>>> (useful to prove the fix is needed)
>>> there is only python testing for the
>>> whole PostgreSQL provider and it is not automatically setup to run.
>> I personally don't think this is an issue. Not everyone has access to
>> a PostGIS setup for this test to run with, and even if they do they
>> may not want running the QGIS test suite to automatically create and
>> modify tables on their setup (eg, I would not be happy if the test was
>> modifying my production database!!)
>> 
>> Again, better test docs would help here.
>> 
>> So in summary - I actually think the current test suite setup is good
>> (and getting constantly better), and the main problem is both lack of
>> testing documentation and training QGIS developers in how to utilise
>> it. I'd be happy to run a session during the hackfest explaining all
>> this, if that's considered worthwhile...
>> 
>> Nyall
> 
> -- 
> Matthias Kuhn
> OPENGIS.ch - https://www.opengis.ch
> Spatial • (Q)GIS • PostGIS • Open Source
> 
> 
> _______________________________________________
> Qgis-psc mailing list
> Qgis-psc at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-psc

Tim Sutton
QGIS Project Steering Committee Member
tim at qgis.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20160220/1804cbc4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.tiff
Type: image/tiff
Size: 9882 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/qgis-psc/attachments/20160220/1804cbc4/attachment.tiff>