[Qgis-developer] Rethinking the testing and release procedure of QGIS

[Martin Dobias]

On Thu, Jul 7, 2011 at 11:32 AM, Tim Sutton <lists at linfiniti.com> wrote:
>
> So there is already a lot of groundwork in place for testing - though
> it has not been maintained for some time for reasons stated in my
> previous email. As mentioned above, Martin once wrote some perl script
> (tests/src/core/test_builder.pl) to automate the generation of test
> stubs.

Just to get the credits right - Tim has written the test builder, I
have never touched it :-)

> I think we should focus on testing the libraries - in the future Qt
> also provides an infrastructure for creating GUI automation tests.
> What I am trying to say really is that we did already put quite some
> thought into testing but we are spread too thin to carry it through.
> So having someone funded to get the tests really working nicely would
> be an awesome addition to the project. From a social side, mandating
> tests hasnt really worked till now - I guess we are always a bit
> scared of losing peoples interest (which we depend on so much) by
> setting too many difficult barriers in place for them to contribute.
> At some point I guess we will have to bite the bullet and start
> refusing contributions unless accompanied by tests (and that they dont
> break existing tests).

I agree with Tim. Basic infrastructure for testing is there, it just
needs quite some work to cover at least commonly used routines in the
core library. At the same time we should reintroduce nightly builds
with testing (and alerts to mailing list on broken tests). There was a
BuildBot instance for qgis on osgeo server, but later something went
wrong with the server and it wasn't restored.

But most importantly we need to teach ourselves to really create unit
tests for newly committed functionality - otherwise we will end up
with broken and incomplete tests like now. Perhaps we need to raise
some kind of challenge :-) And maybe to have someone who will start
shouting when new functionality does not receive tests :-)

Regards
Martin