[GRASS-dev] Test suite for GRASS - proposal, discussion welcome

[Anne Ghisla]

Hello Glynn, all,

On Sun, 2011-05-22 at 20:56 +0100, Glynn Clements wrote:
> Anne Ghisla wrote:
> 
> > Sören and me have discussed today about creating a comprehensive test
> > suite for GRASS. We wrote down the draft [0] and I plan to work on the
> > implementation as soon as possible.
[...]
> > We plan to run unittests and integration tests for both libraries
> > and modules. The test suite will be run after compilation, with a
> > command like:
> >
> >  $ make tests [ proj [ libs | modules | all ] ]
> 
> I'd suggest adding a "test" target to the various *.make files, so you
> can do e.g.:
> 
> 	make -C display test
> 	make -C display/d.rast test

Good idea. Sadly I don't know makefile system well, so most of my
questions are basic. Your suggestion is to explicitely run tests on a
given module. Would it also be possible to run tests in one run,
launching 'make' in the code root folder?

> > Tests are in each module's folder (written in Python)
> 
> Python might be overkill for module tests. The build system absolutely
> requires a Bourne shell (for make), so portability isn't an issue.
> 
> The per-module test script shouldn't need to analyse the output; it
> can just be a sequence of commands which generate textual output (e.g. 
> a test script for a raster module might use r.out.ascii to generate
> this; a test script for a display module would output to a PPM file
> which can just be "cat"d). The analysis would consist of the output
> being compared against a reference file.

It saves us writing lots of functions that would compare datatypes; but
OTOH it would add dependencies of other GRASS modules. Then, it will be
necessary (or at least better) to test first the modules without
dependencies. We would need to check if a test includes more than one
GRASS module... or just let all tests run and fail, and then understand
which modules are actually broken, and causing other (correct) modules'
test to fail.

> E.g. Module.make might have:
> 
> 	$(PGM).out: $(BIN)/$(PGM)$(EXE) $(PGM).test
> 		$(call run_grass,$(SHELL) $(PGM).test > $(PGM).out)
> 	
> 	test: $(PGM).out $(PGM).ok
> 		diff -u $(PGM).out $(PGM).ok || echo "$(PGM) test failed"
> 
> Each directory would have a <module>.test script and a <module>.ok
> reference file.
> 
> This would make it relatively straightfoward to add test cases without
> needing to know Python or the test framework. Just add commands to the
> test script, "make test", and rename <module>.out to <module>.ok.
> 
> More complex features (e.g. multiple test scripts) are possible. The
> general idea remains the same: make the test "system" deal with as
> much as possible so that adding new tests is easy.

I definitely agree, writing tests must be as easy as possible for the
developers. 
We chose Python, as Sören already wrote tests as shell scripts and found
them too cumbersome and difficult to mantain. Python provides already a
good testing framework, and we plan to write base classes specific for
GRASS.
At the moment I'm unable to work on makefiles and have not enough
familiarity with shell to write better tests than the ones written by
Sören; whereas I/we can produce a proof of concept of Python tests in
few weeks.

So, it seems that the choice of the language is up to the coders
involved. If more developers are willing to work in shell, and have some
time to give me some hints, let's go for it. The ultimate goal is to
create a testing framework (currently missing), on which as many
developers as possible can work.

all the best!
Anne
-- 
http://wiki.osgeo.org/wiki/User:Aghisla
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
Url : http://lists.osgeo.org/pipermail/grass-dev/attachments/20110523/d743a891/attachment.bin