Issue #1664: Unit Test support for mapserver

[Frank Warmerdam]

Umberto Nicoletti wrote:
> 'Course I am, as a matter of facts your tests are functional tests (a
> rather different beast from unit tests), while Sean's are unit tests,
> but for python Mapscript.

Umberto,

OK, cool.

> The unit tests I am introducing can go deeper and test the intimate of
> mapserver's code. Also unit tests can test a very limited and well
> defined subset of code, making memory leaks/memory
> misbehaviours/threading issue easier to spot. For instance I have
> written unit tests for the connection pooling which is not covered by
> Sean's tests because connection pooling functions are not exposed by
> mapscript and neither by yours.

I think running valgrind on shp2img or mapserv sessions can
identify most memory leaks.  You are quite right that none of the
existing tests address pooling, or threading issues and it may be
challenging to do this at any level other than the C API (as you
suggest).

> IMHO it is not the case I have to argue in favour of unit tests, but
> quite the contrary, as the beneficial effects of unit testing seem
> pretty obvious to me (and to anybody who's ever used them).

I'm not arguing that there is no benefit to unit tests, and I have
used them.  But if C level unit tests are to reach anything near their
full value, it is necessary to get others involved in extending them
and running them.

> The reasons are:
> - better code
> - safer modifications
> - more confidence
> - regression testing
> - automation of repetitive tasks
> - integration in the build process

I don't see that the above items require another testing
mechanism with the possible exception of things that cannot
practically be unittested via the python mapscript tests, or
functionally tested via msautotest.

> See for instance bug 1629: if we had unit tests we could have
> committed the modifications to 4.8.1 already without having to recall
> why in the first we made those changes in the first place.

I have briefly reviewed Bug 1629 and I cannot see why a python
mapscript or msautotest test item would have been inadequate
to handle it.  Of course, there isn't any such test for PostGIS
support in mapserver.  But we don't need a new approach, we just
need to more fully fill out the existing approach.

>> Currently it seems we have roughly as many test suite mechanisms as we
>> have developers interested in using and supporting them.
>>
>> I imagine you ought to write an RFC proposal on your unittest approach
>> before it is integrated into MapServer.
>>
> 
> If that is necessary I'll do it, but I'd better code than write.

I'm sure we would all rather code than write RFCs, but we decided to
require RFCs for a certain class of changes in MapServer to ensure
we have coherent progress, avoid bloat, and fine tune approaches to
new features before they we are committed to them.

You can certainly maintain a test harnass on your own, but if you
want it in the tree I think it deserves an RFC.  As per ms-rfc-1,
Steve is the final arbiter of what does or does not need an RFC.

Please don't consider my feedback as saying I don't want C unit
tests.  If there is a good case for why we need them instead of
python unit tests, and if you are willing to do the work to establish
them then I'm keen on it.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent