Issue #1664: Unit Test support for mapserver

[Umberto Nicoletti]

Frank,
first of all thanks for your comments.

On 2/14/06, Frank Warmerdam <warmerdam at pobox.com> wrote:
> Umberto Nicoletti wrote:
> > 'Course I am, as a matter of facts your tests are functional tests (a
> > rather different beast from unit tests), while Sean's are unit tests,
> > but for python Mapscript.
>
> Umberto,
>
> OK, cool.
>
> > The unit tests I am introducing can go deeper and test the intimate of
> > mapserver's code. Also unit tests can test a very limited and well
> > defined subset of code, making memory leaks/memory
> > misbehaviours/threading issue easier to spot. For instance I have
> > written unit tests for the connection pooling which is not covered by
> > Sean's tests because connection pooling functions are not exposed by
> > mapscript and neither by yours.
>
> I think running valgrind on shp2img or mapserv sessions can
> identify most memory leaks.  You are quite right that none of the
> existing tests address pooling, or threading issues and it may be
> challenging to do this at any level other than the C API (as you
> suggest).
>
> > IMHO it is not the case I have to argue in favour of unit tests, but
> > quite the contrary, as the beneficial effects of unit testing seem
> > pretty obvious to me (and to anybody who's ever used them).
>
> I'm not arguing that there is no benefit to unit tests, and I have
> used them.  But if C level unit tests are to reach anything near their
> full value, it is necessary to get others involved in extending them
> and running them.

Agreed.

IMO Unit testing is not an opt-in, but a programming approach that
must be used by all developers, that is a requirement.
This is a clear cut case (to quote Tarantino's Jackie Brown) in my
mind: either unit tests get into the repository AND EVERYBODY adopts
them or they don't get into the repository at all.

>
> > The reasons are:
> > - better code
> > - safer modifications
> > - more confidence
> > - regression testing
> > - automation of repetitive tasks
> > - integration in the build process
>

I forgot to add the documentation value that unit tests add to existing code.

> I don't see that the above items require another testing
> mechanism with the possible exception of things that cannot
> practically be unittested via the python mapscript tests, or
> functionally tested via msautotest.
>

Unit tests do not replace functional testing, rather functional tests
complete unit tests.

> > See for instance bug 1629: if we had unit tests we could have
> > committed the modifications to 4.8.1 already without having to recall
> > why in the first we made those changes in the first place.
>
> I have briefly reviewed Bug 1629 and I cannot see why a python
> mapscript or msautotest test item would have been inadequate
> to handle it.  Of course, there isn't any such test for PostGIS
> support in mapserver.  But we don't need a new approach, we just
> need to more fully fill out the existing approach.
>

Of course you can do it with mapscript or msautotest, but that
requires a postgis setup which not everybody has available.
With C unit tests you can just test the query composer function and
verify that the string it produces is the one you are expecting,
dropping entirely the need for a full fledged build environment.
Of course this kind of testing requires refactoring of existing code
to make it more easily testable.
For an example see the two methods I have added to the connection pooling apis:

int  msConnPoolGetConnectionCount();
int  msConnPoolGetConnectionMax();

Best regards,
Umberto

> >> Currently it seems we have roughly as many test suite mechanisms as we
> >> have developers interested in using and supporting them.
> >>
> >> I imagine you ought to write an RFC proposal on your unittest approach
> >> before it is integrated into MapServer.
> >>
> >
> > If that is necessary I'll do it, but I'd better code than write.
>
> I'm sure we would all rather code than write RFCs, but we decided to
> require RFCs for a certain class of changes in MapServer to ensure
> we have coherent progress, avoid bloat, and fine tune approaches to
> new features before they we are committed to them.
>
> You can certainly maintain a test harnass on your own, but if you
> want it in the tree I think it deserves an RFC.  As per ms-rfc-1,
> Steve is the final arbiter of what does or does not need an RFC.
>
> Please don't consider my feedback as saying I don't want C unit
> tests.  If there is a good case for why we need them instead of
> python unit tests, and if you are willing to do the work to establish
> them then I'm keen on it.
>
> Best regards,
> --
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Programmer for Rent
>
>