[GRASS-dev] Test suite for GRASS - proposal, discussion welcome

[Glynn Clements]

Soeren Gebbert wrote:

> I was thinking about a similar approach, but the effort to parse the
> modules XML interface description to identify the command line
> arguments to compare the created data was to much effort for me.

I don't see a need to parse the command; just execute it and see what
files it creates.

> Besides that, the handling of test description, module dependencies
> and the comparison of multiple/timeseries outputs (r.sim.water)
> bothered me too. I still have no simple (interface) answers to this
> issues (maybe these are no issues??).

Dependencies aren't really an issue. You build all of GRASS first,
then test. Any modules which are used for generating test maps or
analysing data are assumed to be correct (they will have test cases of
their own; the most that's required is that such modules are marked as
"critical" so that any failure will be presumed to invalidate the
results of all other tests).

> > E.g. in the above example, if a command creates a raster map named
> > resmap_av and a file named resmap_av.ref exists, that should be
> > sufficient for the framework to deduce that it should compare the map
> > to the reference data using default parameters.
> 
> I see, a simple but powerful approach, i have sometimes the feeling i
> think much to complicated.

I don't normally advocate such approaches, but testing is one of those
areas which (like documentation) is much harder to get people to work
on than e.g. programming, so minimising the effort involved is
important.

Minimising the learning curve is probably even more important. If you
can get people to start writing tests, they're more likely to put in
the effort to learn the less straightforward aspects as it becomes
necessary.

-- 
Glynn Clements <glynn at gclements.plus.com>