[mapserver-dev] the state of msautotest

Alan Boudreault aboudreault at mapgears.com
Wed Sep 12 06:07:17 PDT 2012


I totally agree with you guys. msautotests is in a critical state. :P

I'm aware of 3 tools that seem to be popular these days:

- Buildbot: self-hosted solution in python.

- Jenkins: self-hosted. More complete than buildbot. Nice solution for 
continuous integration and regression tests.

- Travis-CI: web service, current trend with opensource softwares. Seems 
to work very well.  Nice Integration with github projects. (each commit 
are tested)

Unfortunately, I don't know much about them at the moment. The easier 
for sure would be to test Travis-CI and see how it works.

Alan

On 12-09-12 07:14 AM, thomas bonfort wrote:
> After maybe a couple beers too many with Stephan the other day we came
> to talk about mapserver's autotests and it's current deficiencies with
> respect to how we are using them.
>
> To summarize, here are the principle issues that arose:
>
> - The tests aren't run on each commit, which is understandable as it
> takes an additional effort to do so
> - We have a large number of failing tests, which can be attributed to:
>    - the afore-mentioned reason: is a failing test a result of the last
> commit or was it already failing beforehand?
>    - image comparison tests can produce false positives, due to
> different cpu architectures and underlying dependency libraries
> (namely freetype)
>    - image and xml/gml tests can diverge depending on the compile-time
> configuration options that were chosen.
> - we've ended up in a chicken vs. egg situation where tests aren't
> being run because they are failing, and test fail in the long run
> because they are never being used/updated.
>
> To alleviate this, we'd like to set the ball rolling on these ideas:
> - provide a "reference" platform with a kitchen-sink (i.e. compiled
> with all options) mapserver instance. This could be an osgeo ad-hoc vm
> or a dedicated server provided by one of us or our users. (second
> option might be better given the limited resources available at osgeo)
> - go through all the current tests, make sure they pass on this
> reference platform, and update the expected result if not.
> - automatically run the whole test-suite on each commit, and provide a
> web-page or email alert when a test has failed. Does anyone have
> experience with a software solution that could provide that to us?
>
> Thoughts? It's certainly not a miracle solution as it will require
> acting upon when a test starts failing, however we believe that
> automating the runs and removing the false positives will be a nice
> step forward (and give more peace-of-mind to the release-manager when
> packaging a release :) )
>
> cheers,
> thomas
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>


-- 
Alan Boudreault
http://www.mapgears.com/


More information about the mapserver-dev mailing list