[GRASS-dev] Test suite for GRASS - proposal, discussion welcome

Glynn Clements glynn at gclements.plus.com
Mon May 23 10:04:45 EDT 2011


Anne Ghisla wrote:

> > I'd suggest adding a "test" target to the various *.make files, so you
> > can do e.g.:
> > 
> > 	make -C display test
> > 	make -C display/d.rast test
> 
> Good idea. Sadly I don't know makefile system well, so most of my
> questions are basic. Your suggestion is to explicitely run tests on a
> given module. Would it also be possible to run tests in one run,
> launching 'make' in the code root folder?

Yes. In Dir.make (used for directories) the "test" target would be
recursive, i.e. it would run "make -C $dir test" for each
subdirectory, the same way that the "clean" target is handled. This
target would literally be just:

	test: test-recursive

Dir.make already has a pattern rule for "%-recursive", which makes the
stem (whatever the "%" matches) for each subdirectory.

> > > Tests are in each module's folder (written in Python)
> > 
> > Python might be overkill for module tests. The build system absolutely
> > requires a Bourne shell (for make), so portability isn't an issue.
> > 
> > The per-module test script shouldn't need to analyse the output; it
> > can just be a sequence of commands which generate textual output (e.g. 
> > a test script for a raster module might use r.out.ascii to generate
> > this; a test script for a display module would output to a PPM file
> > which can just be "cat"d). The analysis would consist of the output
> > being compared against a reference file.
> 
> It saves us writing lots of functions that would compare datatypes; but
> OTOH it would add dependencies of other GRASS modules. Then, it will be
> necessary (or at least better) to test first the modules without
> dependencies. We would need to check if a test includes more than one
> GRASS module... or just let all tests run and fail, and then understand
> which modules are actually broken, and causing other (correct) modules'
> test to fail.

I don't think that there's any way around this. Few GRASS modules are
useful in isolation. On the positive side, modules which are essential
for tests (e.g. g.region, *.out.ascii) etc are unlikely to be changed
regularly.

As I see it, this isn't really any different to the situation where
someone commits a change which causes a library to fail to compile,
and every module which uses that library also fails to compile.

> We chose Python, as Sören already wrote tests as shell scripts and found
> them too cumbersome and difficult to mantain.

But is this because you're trying to perform analysis of the results
within the test scripts?

-- 
Glynn Clements <glynn at gclements.plus.com>


More information about the grass-dev mailing list