[mapserver-dev] the state of msautotest

[thomas bonfort]

Hi Stephan

On Fri, Sep 14, 2012 at 4:00 PM, Stephan Meißl <stephan at meissl.name> wrote:
> Hi Thomas,
>
> thanks for the credits but as usual you did it in almost no time with
> very little help, simply amazing!
>
> I see you test everything in misc, gdal, renderers, and wxs. What about
> query and also mspython and php? To be honest I've never used the later
> ones but I guess there's a reason why they are there.
Steve is looking into the remaining failing tests in "query", and I
will include those tests in the default runs once it's settled. What's
nice is we even get a postgis instance to install our test data on for
the query tests:
http://travis-ci.org/#!/tbonfort/mapserver/builds/2450204

>
> And then there are also some MapScript tests at least for python and
> java. Is anybody using those? Should we add them to be tested via
> travis?
There are also some php tests that have recently been added.
I'm all for adding other tests in, but I'm not sufficiently into
mapscript myself to take over maintaining them or adding them myself
to the configuration. I will assist with pleasure on irc/email whoever
wants to do so.

We currently have 100% passing tests, and I would like to rely on this
metric in the near future to help assist in the 6.2 release process.
This implies that mapscript tests that are added before then should
pass or in the very least be planned to pass in a very short term.
This means going through all currently failing tests, assessing if
they are real failures or not, and updating the expected output
accordingly. This is important so we can assess that a failed test run
is due to recent changes in the codebase and needs to be acted upon
before our release, rather than an old false positive that no one is
interested in.
In the longer run, I am somewhat hesitant about adding mapscript tests
as mapscript tends to lag a bit behing the development going on in
core mapserver. My main point being that if we allow tests to fail for
some time, we completely loose the advantages of having a system that
informs us immediately of a commit that produced an error. What do you
guys think?



>
> I'm very much in favor of completely adopting it. Question, what happens
> if somebody breaks the build? I vote for something with beers at the
> next FOSS4g :D
that seems like a nice idea, I'm not sure my bank account would be
very pleased about that however :)

regards,
thomas

>
> cu
> Stephan
>
>
> On Thu, 2012-09-13 at 19:00 +0200, thomas bonfort wrote:
>> I have setup a travis account for mapserver and added the necessary
>> files to trigger a test run each time some code is committed to master
>> and branch-6-2. With Stephan's help, we went through the entire
>> test-suite to update the expected results so we can start off with a
>> clean slate, we still however have 10 failing tests that need to be
>> acted upon (the diffs can be seen at
>> http://travis-ci.org/#!/mapserver/mapserver/builds/2437309 then
>> scrolling down to the bottom of the build/test run)
>> I have tried to set the config such that I am the only one getting
>> spammed by failed test logs. If you do receive such emails without
>> having committed any code, please get in touch with me.
>>
>> Hope you like it, this has the potential of becoming a very valuable
>> tool once we iron out the remaining failing tests.
>>
>> regards,
>> thomas
>>
>> On Thu, Sep 13, 2012 at 3:00 PM, Lime, Steve D (DNR)
>> <Steve.Lime at state.mn.us> wrote:
>> > Ditto... I'd actually like to see the tests more tightly integrated in the build
>> > process, so they came with the source and you could do a 'make test'
>> > easily enough.
>> Steve, a "make test" target could be added rather simply now. However
>> the output of the test run will be as error-prone as it was before
>> hand, i.e. you have the risk of ending up with a number of false
>> positives due to your architecture, library versions, and compiled in
>> options.
>> >
>> > Steve
>> >
>> > ________________________________________
>> > From: mapserver-dev-bounces at lists.osgeo.org [mapserver-dev-bounces at lists.osgeo.org] on behalf of Daniel Morissette [dmorissette at mapgears.com]
>> > Sent: Wednesday, September 12, 2012 8:20 AM
>> > To: mapserver-dev at lists.osgeo.org
>> > Subject: Re: [mapserver-dev] the state of msautotest
>> >
>> > I am very supportive of making the msautotests easier to use and more
>> > reliable. Sounds like a few interesting options are on the table already.
>> >
>> > Daniel
>> >
>> > On 12-09-12 7:14 AM, thomas bonfort wrote:
>> >> After maybe a couple beers too many with Stephan the other day we came
>> >> to talk about mapserver's autotests and it's current deficiencies with
>> >> respect to how we are using them.
>> >>
>> >> To summarize, here are the principle issues that arose:
>> >>
>> >> - The tests aren't run on each commit, which is understandable as it
>> >> takes an additional effort to do so
>> >> - We have a large number of failing tests, which can be attributed to:
>> >>    - the afore-mentioned reason: is a failing test a result of the last
>> >> commit or was it already failing beforehand?
>> >>    - image comparison tests can produce false positives, due to
>> >> different cpu architectures and underlying dependency libraries
>> >> (namely freetype)
>> >>    - image and xml/gml tests can diverge depending on the compile-time
>> >> configuration options that were chosen.
>> >> - we've ended up in a chicken vs. egg situation where tests aren't
>> >> being run because they are failing, and test fail in the long run
>> >> because they are never being used/updated.
>> >>
>> >> To alleviate this, we'd like to set the ball rolling on these ideas:
>> >> - provide a "reference" platform with a kitchen-sink (i.e. compiled
>> >> with all options) mapserver instance. This could be an osgeo ad-hoc vm
>> >> or a dedicated server provided by one of us or our users. (second
>> >> option might be better given the limited resources available at osgeo)
>> >> - go through all the current tests, make sure they pass on this
>> >> reference platform, and update the expected result if not.
>> >> - automatically run the whole test-suite on each commit, and provide a
>> >> web-page or email alert when a test has failed. Does anyone have
>> >> experience with a software solution that could provide that to us?
>> >>
>> >> Thoughts? It's certainly not a miracle solution as it will require
>> >> acting upon when a test starts failing, however we believe that
>> >> automating the runs and removing the false positives will be a nice
>> >> step forward (and give more peace-of-mind to the release-manager when
>> >> packaging a release :) )
>> >>
>> >> cheers,
>> >> thomas
>
>
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev