[GRASS-dev] Test suite for GRASS - proposal, discussion welcome

[Glynn Clements]

Soeren Gebbert wrote:

> > message. If any such errors occured as a result of "make test", you
> > would ignore all the other failures. In the same way that if error.log
> > has an error for e.g. lib/gis, you wouldn't bother about all of the
> > subsequent errors and focus on what was wrong with lib/gis.
> 
> Is it possible to stop "make test" in case a library test failed or a
> critical module?

Yes. Although it's not really important. The usual case is that all
tests succeed. If a test fails, it's important that the failure is
reported, but I wouldn't be too concerned about how we handle the
fallout, given that this should be an abnormal situation.

For comparison, the gcc developers have only just come around to the
idea that a failure in locating a #include file shouldn't result in
hundreds of additional errors. Other errors (e.g. mistyping a typedef
or global variable) still cascade.

> > If you're doing repeated tests on a specific module that you're
> > working on, you don't want to be re-running the r.mapcalc, r.out.ascii
> > etc tests every time.
> 
> I don't know if this can be avoided in an automated test system.
> Especially when each time a test case gets executed a temporary mapset
> is created.

Sure it can. "make test" in a module directory runs the tests for that
particular module. Any tests for the test infrastructure itself would
only be done when running "make test" from the top level.

> > I'd export as much precision as is likely to be meaningful, erring on
> > the side of slightly too much precision. The default comparison
> > tolerance should be just large enough that it won't produce noise for
> > the majority of modules. Modules which require more tolerance (e.g.
> > due to numerical instability) should explicitly enlarge the tolerance
> > and/or set an "allowed failures" limit.
> 
> Where to set the precision in the test case? As @precision directive

Yes.

> which will be used for each test in the test case file

You would want to be able to set it for individual commands.

> > Any directory with a Makefile should support "make test" one way or
> > another. Usually via appropriate rules in Lib.make, Module.make,
> > Dir.make, etc. Dir.make would just run "make test" recursively (see
> > the %-recursive pattern rule in Dir.make); the others would look for a
> > test script then use the framework to execute it.
> >
> > The top-level Makefile includes Dir.make, so "make test" would use the
> > recursive rule (a special rule for testing critical modules could be
> > added as a prerequisite). Testing the libraries would just be
> > "make -C lib test" (i.e. recursive in the "lib" directory); similarly
> > for raster, vector, etc.
> 
> Yes. Do we need special rules for critical modules or is the order of
> the module directories in the Makefile's combined with a critical
> annotation sufficient?

The critical annotation is sufficient. "make test" from the top-level
would test everything, including the critical modules. They won't
necessarily be tested first, but they will get tested. It isn't
necessary to optimise the process; failures in critical modules
shouldn't occur often enough for it to matter.

> > I'm not sure we need a separate subdirectory for the test data.
> 
> I am sure we need several. I would suggest a separate test directory
> for each test location: "test_UTM" and "test_LL". Several modules will
> only work in UTM locations, other in booth. Each directory may contain
> several test case files for different modules (r.univar/r3.univar) and
> several .ref files.

This is going to result in the test files outweighing the actual code
by an order of magnitude. A typical module consists of a directory
containing one source file, one Makefile and one HTML file, plus a
.svn subdirectory which takes up more space than the actual module.

Each subdirectory which you add will also have its own .svn
subdirectory. Bear in mind that a typical Linux filesystem uses at
least 4KiB (one x86 page) for each file and directory regardless of
size.

> > One thing that I hadn't thought about much until now is that maps can
> > have a lot of different components, different modules affect different
> > components, and the correct way to perform comparisons would vary
> > between components.
> 
> Do you refer to different feature types in vector maps? Point, line,
> border, area, centroid and so on?

No, I mean in the sense of etc/element_list. Each raster map may have
cellhd, cell, cats, colr, hist, and fcell files, plus a cell_misc
directory containing whatever else (f_format, f_quant, f_range and
null being typical).

E.g. if you're testing r.colors, you need to check the colour table,
not the raster data.

-- 
Glynn Clements <glynn at gclements.plus.com>