[Qgis-developer] Ramping up for 0.8 - a QGIS test framework

[Tim Sutton]

Greetings all

First a quick personal note: Im about to finish my current contract
and will have a 2 month period before starting my next job. In this
period (among other things) I am hoping to spend a good amount of time
on QGIS getting it towards a release ready state. Gary and I have
finished moving off all aspects of the sf project management onto a
private server. This time consuming process accounts for the decided
lack of svn traffic on out part :-( We hope the new project service
will provide us with a good mix of flexibility, reliability, speed of
access and control.

I would really like to address the perception of QGIS being 'buggy'
and 'unstable' through a more rigourous approach to testing and
managing bug reports. I believe there are three things we can do to
address this:

1) Develop a comprehensive unit testing framework
2) Setup automated nightly test builds for validating that head builds
and passes our test suite, along with html reports etc.
3) Setup a system whereby when someone builds QGIS successfully, the
option is offered of posting some information such as OS, distro,
build time, cpu, memory, SVN revision, unit test report etc can be
automatically posted to our server so that we can have a birds eye
view on the status of which environments are 'habitable' for QGIS at
any given time

(Item 3 above is a 'nice to have' for the future)

I propose that for the unit tests we should use the test framework
within qt4. My suggestion is that we create a test stub for every
class and method in the current src/ tree, replicating the same
heirachy under qgis/test/. In addition under /qgis/test/algorithms/ we
can create tests that cut across many classes. Lastly I propose that
we create qgis/test/gui for creating a suite of gui automated tests.
The test stubs would be populated over time as we write more and more
tests.

Ideally these tests should be run as a part of the build process, so
that running 'make' will fail unless all the tests pass too.

The idea would then be to create a test case for every bug (now seems
an especially opportune time since our bug queue has been shortened in
the move over to trac). This will ensure that the all-to-familiar
problem we have of fixed things being rebroken in subsequent commits.

I am volunteering to create the initial test framework structure (and
Ill welcome any help offered). Radim has some excellent ideas for
publicising 0.8 when its released and it would be really nice if we
could feel confident when it goes 'out the door' that the code base is
robust and well tested.

I know there are many ways to skin this cat, so any suggestions /
ideas / concerns / discussion on the matter will welcome. Ill start a
wiki page to document the ideas above more formally when ./ if I get
general approval for carrying out this work....

Best regards to all


--
Tim Sutton

Visit http://qgis.org for a great Open Source GIS
Skype: timlinux
MSN: tim_bdworld at msn.com
Yahoo: tim_bdworld at yahoo.com
Jabber: timlinux
Irc: timlinux on #qgis at freenode.net