[GRASS5] Automated test-system for basic functionalities.

[Sören Gebbert]

On Monday 09 January 2006 12:36, Stephan Holl wrote:
> Hi Developers,
Hi,
> 
> we discussed a automated test-system for modules which
> will report simple OK/error statements for basic functionalities.

Interesting, Markus and i just had the same thoughts and developed a small prototype of a testframework
which tests some GRASS modules of success/failure and try to guess the failure.

www-pool.math.tu-berlin.de/~soeren/grass/modules/GRASS_Testsuite.tar.gz

If you want to test it, do it at your own risk, this is a proof of concept, but it works. Its not well documented.
You have to edit the script RunGRASSTestSuite.sh and add the installation path.
Please run it in spearfish right now, or edit the testregion settings in RunGRASSTestSuite.sh.
I focued to test the r3.* modules, so there are scripts which tests all r3.* modules in grass-6.1-cvs.
But some raster and general tests also available.

> 
> Currently there is no clear idea, but thoughts about a shell-script
> with configuration file have been made.
I think bash shell scripts may be the best choice for this kind of tests, the prototype i developed
is completely written in bash. It is a *nix standart and we are only able to test the success or failure of the modules.
> 
> The dataset is also a problem, since it must be small enough to
> distribute it to testers. Spearfish is probably too big...
I think, the test should be work with every dataset, not only with a provided one. Some tests
i created are producing the input by there own with r.mapcalc or r3.mapcalc.
(ok if r.mapcalc and r3.mapcalc are not working, nothing will be tetested)
And yes, thats not possible for every module.
> 
> A manual example (with spearfish6-data) can be found here[1].
> 
> Perhaps someone on this list can give some ideas, thoughts about
> best-practice of creating such a test-environment?

Tests should be simple to add, even with basic bash knowledge. Every module developer should be able to add tests.
The Testsuite should be easy to extend and it should provide a framework to add simple and extremly, dependenly
sophisticated tests.

There should be a test-directory for every noninteractive command in the sources, which contains the testscripts for this module. 

example:
grass6/raster3/  <----------------- contains all r3.* modules
grass6/raster3/r3.out.vtk <--------------- contains r3.out.vtk sources, description, Makefile and testdierectory
grass6/raster3/r3.out.vtk/testing <--------------- contains r3.out.vtk tests
grass6/raster3/r3.out.vtk/testing/r3.out.vtk-test-simple.sh <------------- simple test file, will run within the testframework, not standalone?!
grass6/raster3/r3.out.vtk/testing/r3.out.vtk-test-complex.sh
...

While installation the tests should be copied into a testsuite directory in the GRASS installation path.

I think the GRASS tests should be run without interaction -> standalone for automatic tests and should produce a parsable complete logfile
with stdout and stderr messages from the tested modules. Also a parsable summary of success and failures should be provided.

Maybe it should produce readable html output, to host a success/failure page at grass.itc.it. So everybody 
will know which module works and which should be fixed. (except the problem is in the lib).

Just my 2c
Best regards
Soeren
> 
> Thank you for input!
> 
> Stephan
> 
> [1] http://mpa.itc.it/markus/mum3/testing_grass6_software.txt
>