[GRASS-dev] Test suite for GRASS - proposal, discussion welcome

[Soeren Gebbert]

Hi,

[snip]
>> It saves us writing lots of functions that would compare datatypes; but
>> OTOH it would add dependencies of other GRASS modules. Then, it will be
>> necessary (or at least better) to test first the modules without
>> dependencies. We would need to check if a test includes more than one
>> GRASS module... or just let all tests run and fail, and then understand
>> which modules are actually broken, and causing other (correct) modules'
>> test to fail.
>
> I don't think that there's any way around this. Few GRASS modules are
> useful in isolation. On the positive side, modules which are essential
> for tests (e.g. g.region, *.out.ascii) etc are unlikely to be changed
> regularly.

Indeed, we will need several modules for data generation, region
settings and data output. The testsuite for grass 6.4 uses g.region,
r/r3/v.out.ascii, r.mapcalc and several other modules.

>> We chose Python, as Sören already wrote tests as shell scripts and found
>> them too cumbersome and difficult to mantain.
>
> But is this because you're trying to perform analysis of the results
> within the test scripts?

In many cases there are situtation where you will need to perform
several independent tests for a single module with several kinds of
outputs each test:

* Module output on stdout
* single or multiple raster and voxel maps
* single or multiple vector maps
* mixed raster and vector output
* and so on

The framework should be able to handle such output automatically i.e:
* parsing the stout and compare it with validation data (text file,
MD5 checksum, ...)
* Generation of ASCII output from raster, vector and voxel maps to
compare with validation data  (text file, MD5 checksum, ...)

To make it as easy as possible the writer of the test should only
provide the command line of the test and data generation and the
validation data. I tried to implement this using the shell approach,
but because of the limitation of the shell this was much to complex. I
am not sure if this kind of data handling can be done using only the
make system?

The next reason is the generation of HTML representation of the test
results for web access. The idea is to run tests on several different
configured server after compilation. The result of these tests will be
posted in the web. This is an important feature, because the developer
will see what modules/libraries failed the tests on which machine
configuration. Have a look at the VTK dashboard[1] and the result of
the old testsuite [2].

The old testsuite has the possibility to run each module in a valgrind
environment to find memory leaks. It is able to validate outputs of
different kind against MD5 checksums. It logs stdout and stderr of the
module and provides this data as HTML output. [3] It links to the test
description [4].

Using python the implementation will be much easier as with shell and
the related unix toolset.

Best regards
Soeren

[1] http://www.cdash.org/CDash/index.php?project=VTK
[2] http://www-pool.math.tu-berlin.de/~soeren/grass/GRASS_TestSuite/html_grass-6.4/
[3] http://www-pool.math.tu-berlin.de/~soeren/grass/GRASS_TestSuite/html_grass-6.4/log.html
[4] http://www-pool.math.tu-berlin.de/~soeren/grass/GRASS_TestSuite/html_grass-6.4//r.gwflow-test.sh.html

>
> --
> Glynn Clements <glynn at gclements.plus.com>
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev
>