[mapserver-dev] c unit tests in MapServer

[Paul Ramsey]

Once the tester is set up, it's pretty easy, IMO. The PostGIS setup is
basically the cu_tester application with all the cunit-specific and
project-specific code (like our hooking into our error handlers), then
all the test suites, which are very simple.

https://github.com/postgis/postgis/blob/svn-trunk/liblwgeom/cunit/cu_tester.c

Most of the code there is actually just stuff to allow users to choose
a particular suite or test to run, rather than running all 1000+
(useful when debugging a particular test)

Everything else is just the tests themselves, which are pretty simple.

https://github.com/postgis/postgis/blob/svn-trunk/liblwgeom/cunit/cu_in_wkt.c

Since much of the logic in Mapserver is in libmapserver now, cunit
could be a pretty good fit for testing that piece, in the same way
that it tests just the liblwgeom piece of PostGIS.

P.


On Mon, Dec 8, 2014 at 8:07 AM, Lime, Steve D (MNIT)
<Steve.Lime at state.mn.us> wrote:
> How bad was it to get started? A quick peek at the CUnit simple example didn't seem that simple.
>
> -----Original Message-----
> From: Paul Ramsey [mailto:pramsey at cleverelephant.ca]
> Sent: Saturday, December 06, 2014 5:33 PM
> To: Lime, Steve D (MNIT)
> Cc: geographika at gmail.com; mapserver-dev at lists.osgeo.org
> Subject: Re: [mapserver-dev] c unit tests in MapServer
>
> We ended up with belt *and* suspenders in PostGIS, and it works OK. We have an online regression test, which actually runs the data base and slams SQL through it, which tests for correctness in the large. But we also have a cunit hardness that tests the underlying library, things like computational geometry, format parsers, etc, in isolation from the database backend. Once a code base gets this far along, and is this mature, adding coverage is perhaps "too little, too late", but once you have coverage it's hard to imagine working without it.
>
> P.
>
> On Thu, Dec 4, 2014 at 12:34 PM, Lime, Steve D (MNIT) <Steve.Lime at state.mn.us> wrote:
>> First, thanks so much for your willingness to report/research/fix bugs
>> – it’s invaluable.
>>
>> I don’t know that we’d need to make the full leap to a C Unit testing
>> framework to make use of them on a ticket by ticket basis. If you
>> wanted to roll a simple C program that tests just the function in
>> question I’d certainly want to use it. It makes sense in this case. So
>> often though bugs seem to involve a series of functions performing a
>> complex operation where a more complete configuration (e.g.
>> data+mapfile) and  request is necessary to trigger the issue.
>>
>>
>>
>> Steve
>>
>>
>>
>> From: mapserver-dev-bounces at lists.osgeo.org
>> [mailto:mapserver-dev-bounces at lists.osgeo.org] On Behalf Of
>> geographika
>> Sent: Wednesday, December 03, 2014 8:56 AM
>> To: mapserver-dev at lists.osgeo.org
>> Subject: [mapserver-dev] c unit tests in MapServer
>>
>>
>>
>> Hi list,
>>
>>
>> With almost 20 years of development, the MapServer codebase can be
>> overwhelming at first. For most open source projects I've found the
>> easiest way into helping with a project is:
>>
>>
>>
>> 1.       Reporting bugs
>>
>> 2.       Finding the problematic code
>>
>> 3.       Submitting a patch and unit test.
>>
>>
>>
>> In 2006 there was a Request For Comments (#12) by Umberto Nicoletti
>> about adding C Unit tests:
>>
>>
>>
>> http://www.mapserver.org/development/rfc/ms-rfc-12.html
>>
>> Also see GitHub issue #1664:
>> https://github.com/mapserver/mapserver/issues/1664
>>
>> Obviously a full test suite of existing code isn't possible, however
>> if the testing framework and skeleton folder/file structure was set up
>> it may help more concrete contributions from developers outside of the
>> core MapServer team.
>>
>>
>>
>> There have been a few bugs I've submitted where I would feel
>> comfortable adding a unit test to an existing file to show the error,
>> but creating full msautotest examples seems very daunting. As a
>> concrete example while testing the latest dev build I noticed the
>> msParseTime function in the maptime.c module threw an error. Unit
>> tests for this function seem like the perfect solution for these types
>> of bugs, and to avoid breaking changes in the future.
>>
>>
>>
>> https://github.com/mapserver/mapserver/commit/0cc47e3027a75cf18c531069
>> 4877999c6b245371
>>
>>
>>
>> C Unit tests have come up a few times in the past e.g. Paul Ramsey's
>> post about PostGIS driver testing:
>> http://osgeo-org.1560.x6.nabble.com/cunit-RFC-12-td4261340.html
>>
>>
>>
>> Frank Warmerdam's response (in 2008) was -
>>
>>
>>
>> My preference remains to test specific functions as part of a larger
>>
>> operations since we already have such technology in place (ie.
>> msautotest)
>>
>> or to expose the functionality through mapscript and use mapscript
>> tests
>>
>> like Sean's python unit tests.
>>
>>
>>
>> I do not feel that introducing another testing methodology into the
>>
>> project would be wise.
>>
>>
>>
>> The Python unit tests are again useful, but require testing against
>> full builds, a difficult task for those not familiar with make files
>> and gathering dependencies.
>>
>>  Testing database drivers as I understand it would require a full test
>> database setup with msautotest, whereas for many bugs it is clear
>> which function needs modifying and a unit test could be added quickly
>> to highlight and help fix the bug.
>>
>>
>>
>> I've only experience in Python and .NET unit testing so perhaps C
>> testing is more trouble than it is worth.
>> I guess it is up to the core devs whether it has any benefit to
>> themselves, or the project as a whole, whether this issue should be reconsidered.
>>
>>
>>
>> Regards,
>>
>> Seth
>>
>>
>>
>> --
>> web: http://geographika.co.uk
>> twitter: @geographika
>>
>>
>>
>>
>> _______________________________________________
>> mapserver-dev mailing list
>> mapserver-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev