[Qgis-developer] Rethinking the testing and release procedure of QGIS

[Sandro Santilli]

On Thu, Jul 07, 2011 at 05:23:28PM +0200, Marco Hugentobler wrote:

> Being in favor of unit tests, we cannot hope to solve with them all the 
> stability problems, as some mails here suggest. It is just one small piece.

I wouldn't call it small. But it surely won't easily test each and every
possible combination of interactions. Not deterministically at least
(there are some testsuites aimed at throwing random inputs to tools, to
catch a wider set of bugs w/out tight control over the premises, postgis
garden test is one, we've used zzuf for gnash).

> In fact, thinking about bugfixes in the past, e.g. for the 1.7 release,
> most of them occured as a complex combination of several factors and user
> interaction, very hard to detect with unit tests.

This may be a limit on the currently available tools for testing GUIs,
but anyway nobody said it'd be an easy task.

> Another point that needs to be considered is that the unit testing code needs 
> to be maintained. Adding a test for every little change, even bugfix, would 
> create a huge amount of testing code and sample datasets.

The bigger the testsuite, the safer the codebase !

What has to be maintained about testcases ?
The only reason to change a testcase would be a change in the underlying
classes interfaces, which is one of the causes for instability, so having
testcases that break might be an aim to keep APIs stable.

> What about alternatively creating a unit test base that is limited in size
> but well maintained, e.g. a good coverage of the core classes? 

I don't see any testing effort being exclusive. The more testing
means, the better.

--strk;

  ()   Free GIS & Flash consultant/developer
  /\   http://strk.keybit.net/services.html