[GRASS-dev] Test suite for GRASS - proposal, discussion welcome

[Soeren Gebbert]

Hi Glynn,

2011/6/3 Glynn Clements <glynn at gclements.plus.com>:
>
> Soeren Gebbert wrote:
>
>> i have updated the wiki and added a simple Python example of a
>> hypothetical r.series test. Please have a look at:
>> http://grass.osgeo.org/wiki/Test_Suite#Test_framework
>>
>> The sample Python code shows in principle how a test case would look
>> like and what kind of methods should be provided by the framework.
>
> This isn't a particularly good example. For most modules, you won't be
> able to synthesise the reference data with r.mapcalc; you'll need to
> provide it.

Indeed, you are right, bad example. I just tried to show some
hypothetical interface methods. There should be of course default
region settings and test maps available.

> But my main concern is that few people will bother to learn a complex
> test framework. If you want people to write tests, they need to be
> little more than sample commands. Anything else (e.g. directives which
> indicate how to compare the output) needs to be capable of being
> summarised in a dozen lines. E.g.:
>
>        # Test the average method
>        r.series input=testmap1,testmap2 output=resmap_av method=average
>        # Test the sum method
>        r.series input=testmap1,testmap2 output=resmap_sum method=sum
>        # Test the max method
>        r.series input=testmap1,testmap2 output=resmap_max method=max

I was thinking about a similar approach, but the effort to parse the
modules XML interface description to identify the command line
arguments to compare the created data was to much effort for me.

Besides that, the handling of test description, module dependencies
and the comparison of multiple/timeseries outputs (r.sim.water)
bothered me too. I still have no simple (interface) answers to this
issues (maybe these are no issues??).

> The implementation of the framework which executes test scripts can be
> as complex as you like, but the interface (i.e. the syntax of test
> scripts) needs to be trivial.

Well, i thought my example is trivial ... but your may be right. My
perspective is a developers perspective familiar with different test
frameworks. Switching to a software tester perspective is difficult
for me, so many thanks for your feedback Glynn!

> Boilerplate should be minimised; a region should already be set, test
> maps should be available, parameters should have default values. Don't
> force the user to specify anything that the framework ought to be able
> to do itself.

Your are completely right, i had this in mind too. I would like to put
several small test locations in the grass source tree for this reason.
These locations, which differ only in the coordinate reference system,
should provide test maps and data for most test cases in the PERMANENT
mapset. The module test mapsets are created on the fly and removed for
each module test, to avoid a test location mess-up.

> E.g. in the above example, if a command creates a raster map named
> resmap_av and a file named resmap_av.ref exists, that should be
> sufficient for the framework to deduce that it should compare the map
> to the reference data using default parameters.

I see, a simple but powerful approach, i have sometimes the feeling i
think much to complicated.

Best regards
Soeren

>
> --
> Glynn Clements <glynn at gclements.plus.com>
>