[mapserver-dev] the state of msautotest

[Daniel Morissette]

I am very supportive of making the msautotests easier to use and more 
reliable. Sounds like a few interesting options are on the table already.

Daniel

On 12-09-12 7:14 AM, thomas bonfort wrote:
> After maybe a couple beers too many with Stephan the other day we came
> to talk about mapserver's autotests and it's current deficiencies with
> respect to how we are using them.
>
> To summarize, here are the principle issues that arose:
>
> - The tests aren't run on each commit, which is understandable as it
> takes an additional effort to do so
> - We have a large number of failing tests, which can be attributed to:
>    - the afore-mentioned reason: is a failing test a result of the last
> commit or was it already failing beforehand?
>    - image comparison tests can produce false positives, due to
> different cpu architectures and underlying dependency libraries
> (namely freetype)
>    - image and xml/gml tests can diverge depending on the compile-time
> configuration options that were chosen.
> - we've ended up in a chicken vs. egg situation where tests aren't
> being run because they are failing, and test fail in the long run
> because they are never being used/updated.
>
> To alleviate this, we'd like to set the ball rolling on these ideas:
> - provide a "reference" platform with a kitchen-sink (i.e. compiled
> with all options) mapserver instance. This could be an osgeo ad-hoc vm
> or a dedicated server provided by one of us or our users. (second
> option might be better given the limited resources available at osgeo)
> - go through all the current tests, make sure they pass on this
> reference platform, and update the expected result if not.
> - automatically run the whole test-suite on each commit, and provide a
> web-page or email alert when a test has failed. Does anyone have
> experience with a software solution that could provide that to us?
>
> Thoughts? It's certainly not a miracle solution as it will require
> acting upon when a test starts failing, however we believe that
> automating the runs and removing the false positives will be a nice
> step forward (and give more peace-of-mind to the release-manager when
> packaging a release :) )
>
> cheers,
> thomas
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>


-- 
Daniel Morissette
http://www.mapgears.com/
Provider of Professional MapServer Support since 2000