[mapserver-dev] msautotest considered worthless?

[Tamas Szekeres]

2008/4/30 Frank Warmerdam <warmerdam at pobox.com>:
>
>  I will note, today we observed one technique for making use of
>  the autotest suite.  That is to delete all the expected result
>  files, and regenerate them locally.  Then as you change mapserver,
>  any future failures should be worth investigating.
>
>  Wiping and generating local expected results can be accomplished like
>  this:
>
>   cd .../msautotest
>   rm */expected/*
>   (cd gdal ; run_test.py)
>   (cd misc ; run_test.py)
>   (cd wxs ; run_test.py)
>

The only concern I have is that one might be uncertain of the
correctness of the results in the first turn, either . I'd rather
support a mechanism to pick up the desired results from different
directories depending on the platform, we could probably use an
environment setting so as to point to the proper place, or the
.autotest script could query the platform and set up the path
accordingly.

Currently most of the buildbot instances support inspecting the
results at the web interface. If the author makes sure about the
correctness, then he could download the result and commit back to the
repository. Or I voulteer to commit the new result files regularly.

One other thing is that the scripts should return with error in case
of failures so the buildbot could report those tests correctly as
errors.


Best regards,

Tamas