[GRASS-dev] Test suite for GRASS - proposal, discussion welcome

[Glynn Clements]

Anne Ghisla wrote:

> Sören and me have discussed today about creating a comprehensive test
> suite for GRASS. We wrote down the draft [0] and I plan to work on the
> implementation as soon as possible.
> 
> I hope it is of interest of the developers team, and of course more
> opinions are welcome, as well as coding hands.

A couple of comments (I thought about using the Wiki "talk" page but I
wasn't sure if anyone would read it):

> We plan to run unittests and integration tests for both libraries
> and modules. The test suite will be run after compilation, with a
> command like:
>
>  $ make tests [ proj [ libs | modules | all ] ]

I'd suggest adding a "test" target to the various *.make files, so you
can do e.g.:

	make -C display test
	make -C display/d.rast test

etc.

> Tests are in each module's folder (written in Python)

Python might be overkill for module tests. The build system absolutely
requires a Bourne shell (for make), so portability isn't an issue.

The per-module test script shouldn't need to analyse the output; it
can just be a sequence of commands which generate textual output (e.g. 
a test script for a raster module might use r.out.ascii to generate
this; a test script for a display module would output to a PPM file
which can just be "cat"d). The analysis would consist of the output
being compared against a reference file.

E.g. Module.make might have:

	$(PGM).out: $(BIN)/$(PGM)$(EXE) $(PGM).test
		$(call run_grass,$(SHELL) $(PGM).test > $(PGM).out)
	
	test: $(PGM).out $(PGM).ok
		diff -u $(PGM).out $(PGM).ok || echo "$(PGM) test failed"

Each directory would have a <module>.test script and a <module>.ok
reference file.

This would make it relatively straightfoward to add test cases without
needing to know Python or the test framework. Just add commands to the
test script, "make test", and rename <module>.out to <module>.ok.

More complex features (e.g. multiple test scripts) are possible. The
general idea remains the same: make the test "system" deal with as
much as possible so that adding new tests is easy.

-- 
Glynn Clements <glynn at gclements.plus.com>