[mapserver-dev] the state of msautotest

[Stephan Meißl]

On Wed, 2012-09-12 at 13:14 +0200, thomas bonfort wrote:
> After maybe a couple beers too many with Stephan the other day we came
> to talk about mapserver's autotests and it's current deficiencies with
> respect to how we are using them.
> 
> To summarize, here are the principle issues that arose:
> 
> - The tests aren't run on each commit, which is understandable as it
> takes an additional effort to do so
> - We have a large number of failing tests, which can be attributed to:
>   - the afore-mentioned reason: is a failing test a result of the last
> commit or was it already failing beforehand?
>   - image comparison tests can produce false positives, due to
> different cpu architectures and underlying dependency libraries
> (namely freetype)
>   - image and xml/gml tests can diverge depending on the compile-time
> configuration options that were chosen.
> - we've ended up in a chicken vs. egg situation where tests aren't
> being run because they are failing, and test fail in the long run
> because they are never being used/updated.
> 
> To alleviate this, we'd like to set the ball rolling on these ideas:
> - provide a "reference" platform with a kitchen-sink (i.e. compiled
> with all options) mapserver instance. This could be an osgeo ad-hoc vm
> or a dedicated server provided by one of us or our users. (second
> option might be better given the limited resources available at osgeo)
> - go through all the current tests, make sure they pass on this
> reference platform, and update the expected result if not.
> - automatically run the whole test-suite on each commit, and provide a
> web-page or email alert when a test has failed. Does anyone have
> experience with a software solution that could provide that to us?
> 
> Thoughts? It's certainly not a miracle solution as it will require
> acting upon when a test starts failing, however we believe that
> automating the runs and removing the false positives will be a nice
> step forward (and give more peace-of-mind to the release-manager when
> packaging a release :) )
> 
> cheers,
> thomas


Thomas, All,

thanks for the great writeup - even after a couple of beers :)

Meanwhile I had a look at Travis-CI and its features like direct GitHub
integration, continuous integration even on pull requests, hosting, etc.
look quite promising. We already started to have a more closer look at
it using our fork. We'll keep you informed who thinks are progressing.

cu
Stephan

[1] http://travis-ci.org/