[Qgis-developer] Rethinking the testing and release procedure of QGIS

[Marco Hugentobler]

> What has to be maintained about testcases ?
> The only reason to change a testcase would be a change in the underlying
> classes interfaces, which is one of the causes for instability, so having
> testcases that break might be an aim to keep APIs stable.

I can think of several reason why the testcase need changes and the 
maintainance is not 0 (usually there are even more cases in practice):

- The dependencies of QGIS do change (both in API and in behaviour). Most 
tests use Qt classes. The Qt API _and_ behaviour can change. Anybody remembers 
the switch from Qt3 to Qt4? It was a huge effort to port all the classes (the 
bigger the testsuite, the more changes).

- The API of QGIS might indeed change. E.g. say Martin implements a new 
redesigned symbology engine -> he needs to adapt all the symbology related 
test (the more the better?)

- The 'no change without unit test' policy means also test for internal 
classes (not in the public API)


Marco







Am Donnerstag, 7. Juli 2011, 17.53:53 schrieb Sandro Santilli:
> On Thu, Jul 07, 2011 at 05:23:28PM +0200, Marco Hugentobler wrote:
> > Being in favor of unit tests, we cannot hope to solve with them all the
> > stability problems, as some mails here suggest. It is just one small
> > piece.
> 
> I wouldn't call it small. But it surely won't easily test each and every
> possible combination of interactions. Not deterministically at least
> (there are some testsuites aimed at throwing random inputs to tools, to
> catch a wider set of bugs w/out tight control over the premises, postgis
> garden test is one, we've used zzuf for gnash).
> 
> > In fact, thinking about bugfixes in the past, e.g. for the 1.7 release,
> > most of them occured as a complex combination of several factors and user
> > interaction, very hard to detect with unit tests.
> 
> This may be a limit on the currently available tools for testing GUIs,
> but anyway nobody said it'd be an easy task.
> 
> > Another point that needs to be considered is that the unit testing code
> > needs to be maintained. Adding a test for every little change, even
> > bugfix, would create a huge amount of testing code and sample datasets.
> 
> The bigger the testsuite, the safer the codebase !
> 
> What has to be maintained about testcases ?
> The only reason to change a testcase would be a change in the underlying
> classes interfaces, which is one of the causes for instability, so having
> testcases that break might be an aim to keep APIs stable.
> 
> > What about alternatively creating a unit test base that is limited in
> > size but well maintained, e.g. a good coverage of the core classes?
> 
> I don't see any testing effort being exclusive. The more testing
> means, the better.
> 
> --strk;
> 
>   ()   Free GIS & Flash consultant/developer
>   /\   http://strk.keybit.net/services.html


-- 
Dr. Marco Hugentobler
Sourcepole - Linux & Open Source Solutions
Churerstr. 22, CH-8808 Pfäffikon SZ, Switzerland
marco.hugentobler at sourcepole.ch http://www.sourcepole.ch
Technical Advisor QGIS Project Steering Committee