[mapserver-dev] msautotest considered worthless?

Frank Warmerdam warmerdam at pobox.com
Tue Apr 29 21:58:59 EDT 2008

Paul Ramsey wrote:
> After my dance-of-the-thousand-commits with the buildbots last
> weekend, I pulled down the autotest suite to see if regression testing
> could have helped me avoid the Theatre Of Pain.
> What I found was that, to my surprise, huge numbers of the tests in
> trunk against trunk fail. I installed the perceptual diff and there
> are still large numbers of failures. My thought is that a regression
> suite, in order to be useful, should be committed in a state such that
> it always has *no* failures, so that the existence of a failure in
> your local sandbox is a positive Call to Action, to figure out What
> You Did Wrong.
> So, am I missing something, or has the test suite basically been
> allowed to degenerate into a heap?  There is no point in having the
> suite if it is not maintained and used along with the code, is the
> appetite basically not there?


As we discussed in IRC there are two issues at play.

One is that image results from the test suite seem to vary in different
builds for reasons that are not clear, nor easily avoided.  I suspect
that freetype version, compiler switches, and operating system all play
a factor in perturbing results.

This results in lots of "false failures" on machines other than the
reference platform from which the expected results were committed.

The false failures in turn frustrating those who might otherwise
use and extend msautotest.  This results in changes being made in
mapserver without the tests being updated, resulting in actual

I don't know a real solution to this.

I will note, today we observed one technique for making use of
the autotest suite.  That is to delete all the expected result
files, and regenerate them locally.  Then as you change mapserver,
any future failures should be worth investigating.

Wiping and generating local expected results can be accomplished like

   cd .../msautotest
   rm */expected/*
   (cd gdal ; run_test.py)
   (cd misc ; run_test.py)
   (cd wxs ; run_test.py)

Best regards,
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

More information about the mapserver-dev mailing list