[mapserver-dev] the state of msautotest

[thomas bonfort]

I have setup a travis account for mapserver and added the necessary
files to trigger a test run each time some code is committed to master
and branch-6-2. With Stephan's help, we went through the entire
test-suite to update the expected results so we can start off with a
clean slate, we still however have 10 failing tests that need to be
acted upon (the diffs can be seen at
http://travis-ci.org/#!/mapserver/mapserver/builds/2437309 then
scrolling down to the bottom of the build/test run)
I have tried to set the config such that I am the only one getting
spammed by failed test logs. If you do receive such emails without
having committed any code, please get in touch with me.

Hope you like it, this has the potential of becoming a very valuable
tool once we iron out the remaining failing tests.

regards,
thomas

On Thu, Sep 13, 2012 at 3:00 PM, Lime, Steve D (DNR)
<Steve.Lime at state.mn.us> wrote:
> Ditto... I'd actually like to see the tests more tightly integrated in the build
> process, so they came with the source and you could do a 'make test'
> easily enough.
Steve, a "make test" target could be added rather simply now. However
the output of the test run will be as error-prone as it was before
hand, i.e. you have the risk of ending up with a number of false
positives due to your architecture, library versions, and compiled in
options.
>
> Steve
>
> ________________________________________
> From: mapserver-dev-bounces at lists.osgeo.org [mapserver-dev-bounces at lists.osgeo.org] on behalf of Daniel Morissette [dmorissette at mapgears.com]
> Sent: Wednesday, September 12, 2012 8:20 AM
> To: mapserver-dev at lists.osgeo.org
> Subject: Re: [mapserver-dev] the state of msautotest
>
> I am very supportive of making the msautotests easier to use and more
> reliable. Sounds like a few interesting options are on the table already.
>
> Daniel
>
> On 12-09-12 7:14 AM, thomas bonfort wrote:
>> After maybe a couple beers too many with Stephan the other day we came
>> to talk about mapserver's autotests and it's current deficiencies with
>> respect to how we are using them.
>>
>> To summarize, here are the principle issues that arose:
>>
>> - The tests aren't run on each commit, which is understandable as it
>> takes an additional effort to do so
>> - We have a large number of failing tests, which can be attributed to:
>>    - the afore-mentioned reason: is a failing test a result of the last
>> commit or was it already failing beforehand?
>>    - image comparison tests can produce false positives, due to
>> different cpu architectures and underlying dependency libraries
>> (namely freetype)
>>    - image and xml/gml tests can diverge depending on the compile-time
>> configuration options that were chosen.
>> - we've ended up in a chicken vs. egg situation where tests aren't
>> being run because they are failing, and test fail in the long run
>> because they are never being used/updated.
>>
>> To alleviate this, we'd like to set the ball rolling on these ideas:
>> - provide a "reference" platform with a kitchen-sink (i.e. compiled
>> with all options) mapserver instance. This could be an osgeo ad-hoc vm
>> or a dedicated server provided by one of us or our users. (second
>> option might be better given the limited resources available at osgeo)
>> - go through all the current tests, make sure they pass on this
>> reference platform, and update the expected result if not.
>> - automatically run the whole test-suite on each commit, and provide a
>> web-page or email alert when a test has failed. Does anyone have
>> experience with a software solution that could provide that to us?
>>
>> Thoughts? It's certainly not a miracle solution as it will require
>> acting upon when a test starts failing, however we believe that
>> automating the runs and removing the false positives will be a nice
>> step forward (and give more peace-of-mind to the release-manager when
>> packaging a release :) )
>>
>> cheers,
>> thomas
>> _______________________________________________
>> mapserver-dev mailing list
>> mapserver-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>
>
>
> --
> Daniel Morissette
> http://www.mapgears.com/
> Provider of Professional MapServer Support since 2000
>
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>
>
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev