[mapserver-dev] Re: Automated test suite?

[Frank Warmerdam]

Daniel Morissette wrote:
> Frank Warmerdam wrote:
> 
>>I am also constructing an extensive test suite for various raster issues
>>that is small, and can be run with just shp2img by a script, and automatically
>>compared to expected results.  I hope to check this in when completed.
>>
>>Is there any other automated testsuites for MapServer at this time?
>>
> 
> 
> I believe Steve has a test suite (a set of apps to walk through), but
> I don't think it's automated... I'll CC: mapserver-dev so that he can
> comment.
> 
> Personally I think an automated suite would be very welcome.  I've had
> this in mind for quite a while and I would probably add tests to it for
> OGR and WMS support at least.  You wrote that you would commit your
> tests to CVS, but should it not be a separate CVS project to avoid
> bloating the MapServer source tree.
> 
> BTW, how do you do your validation of the result?  Do you compare images
> pixel by pixel?  I've always been worried that just changing the version
> of a lib like GD could be enough to make a difference of a few pixels
> and break the tests?  What do you think?

Daniel,

I do a binary comparison of the output image to an expected result file.
They have to match exactly.  The tests will be fairly sensitive to
changes to rendering in stuff like GD but I am not sure what the
alternatives are.  Note that on failure the tester can then visually
compare the result, and the expected result and decide if the differences
are material.

We may learn more about this as we go.

By the way, where should I check in the tests? I agree it doesn't belong
in the source tree, but luckily the tests aren't very large.  I deliberately
have created tiny test files and small outputs to avoid too much bloat.

Best regards,

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent