[mapserver-dev] the state of msautotest

[thomas bonfort]

After maybe a couple beers too many with Stephan the other day we came
to talk about mapserver's autotests and it's current deficiencies with
respect to how we are using them.

To summarize, here are the principle issues that arose:

- The tests aren't run on each commit, which is understandable as it
takes an additional effort to do so
- We have a large number of failing tests, which can be attributed to:
  - the afore-mentioned reason: is a failing test a result of the last
commit or was it already failing beforehand?
  - image comparison tests can produce false positives, due to
different cpu architectures and underlying dependency libraries
(namely freetype)
  - image and xml/gml tests can diverge depending on the compile-time
configuration options that were chosen.
- we've ended up in a chicken vs. egg situation where tests aren't
being run because they are failing, and test fail in the long run
because they are never being used/updated.

To alleviate this, we'd like to set the ball rolling on these ideas:
- provide a "reference" platform with a kitchen-sink (i.e. compiled
with all options) mapserver instance. This could be an osgeo ad-hoc vm
or a dedicated server provided by one of us or our users. (second
option might be better given the limited resources available at osgeo)
- go through all the current tests, make sure they pass on this
reference platform, and update the expected result if not.
- automatically run the whole test-suite on each commit, and provide a
web-page or email alert when a test has failed. Does anyone have
experience with a software solution that could provide that to us?

Thoughts? It's certainly not a miracle solution as it will require
acting upon when a test starts failing, however we believe that
automating the runs and removing the false positives will be a nice
step forward (and give more peace-of-mind to the release-manager when
packaging a release :) )

cheers,
thomas