Issue #1664: Unit Test support for mapserver

[Umberto Nicoletti]

I have published a first public draft of rfc-12: Unit tests

http://mapserver.gis.umn.edu/development/rfc/ms-rfc-12

Please share your thoughts.

Regards,
Umberto


On 2/16/06, Umberto Nicoletti <umberto.nicoletti at gmail.com> wrote:
> I am trying to add an rfc, could someone enable rfc editing on the
> plone site for my account (unicoletti) or explain how to write one?
>
> Regards,
> Umberto
>
> On 2/15/06, Umberto Nicoletti <umberto.nicoletti at gmail.com> wrote:
> > Frank,
> > first of all thanks for your comments.
> >
> > On 2/14/06, Frank Warmerdam <warmerdam at pobox.com> wrote:
> > > Umberto Nicoletti wrote:
> > > > 'Course I am, as a matter of facts your tests are functional tests (a
> > > > rather different beast from unit tests), while Sean's are unit tests,
> > > > but for python Mapscript.
> > >
> > > Umberto,
> > >
> > > OK, cool.
> > >
> > > > The unit tests I am introducing can go deeper and test the intimate of
> > > > mapserver's code. Also unit tests can test a very limited and well
> > > > defined subset of code, making memory leaks/memory
> > > > misbehaviours/threading issue easier to spot. For instance I have
> > > > written unit tests for the connection pooling which is not covered by
> > > > Sean's tests because connection pooling functions are not exposed by
> > > > mapscript and neither by yours.
> > >
> > > I think running valgrind on shp2img or mapserv sessions can
> > > identify most memory leaks.  You are quite right that none of the
> > > existing tests address pooling, or threading issues and it may be
> > > challenging to do this at any level other than the C API (as you
> > > suggest).
> > >
> > > > IMHO it is not the case I have to argue in favour of unit tests, but
> > > > quite the contrary, as the beneficial effects of unit testing seem
> > > > pretty obvious to me (and to anybody who's ever used them).
> > >
> > > I'm not arguing that there is no benefit to unit tests, and I have
> > > used them.  But if C level unit tests are to reach anything near their
> > > full value, it is necessary to get others involved in extending them
> > > and running them.
> >
> > Agreed.
> >
> > IMO Unit testing is not an opt-in, but a programming approach that
> > must be used by all developers, that is a requirement.
> > This is a clear cut case (to quote Tarantino's Jackie Brown) in my
> > mind: either unit tests get into the repository AND EVERYBODY adopts
> > them or they don't get into the repository at all.
> >
> > >
> > > > The reasons are:
> > > > - better code
> > > > - safer modifications
> > > > - more confidence
> > > > - regression testing
> > > > - automation of repetitive tasks
> > > > - integration in the build process
> > >
> >
> > I forgot to add the documentation value that unit tests add to existing code.
> >
> > > I don't see that the above items require another testing
> > > mechanism with the possible exception of things that cannot
> > > practically be unittested via the python mapscript tests, or
> > > functionally tested via msautotest.
> > >
> >
> > Unit tests do not replace functional testing, rather functional tests
> > complete unit tests.
> >
> > > > See for instance bug 1629: if we had unit tests we could have
> > > > committed the modifications to 4.8.1 already without having to recall
> > > > why in the first we made those changes in the first place.
> > >
> > > I have briefly reviewed Bug 1629 and I cannot see why a python
> > > mapscript or msautotest test item would have been inadequate
> > > to handle it.  Of course, there isn't any such test for PostGIS
> > > support in mapserver.  But we don't need a new approach, we just
> > > need to more fully fill out the existing approach.
> > >
> >
> > Of course you can do it with mapscript or msautotest, but that
> > requires a postgis setup which not everybody has available.
> > With C unit tests you can just test the query composer function and
> > verify that the string it produces is the one you are expecting,
> > dropping entirely the need for a full fledged build environment.
> > Of course this kind of testing requires refactoring of existing code
> > to make it more easily testable.
> > For an example see the two methods I have added to the connection pooling apis:
> >
> > int  msConnPoolGetConnectionCount();
> > int  msConnPoolGetConnectionMax();
> >
> > Best regards,
> > Umberto
> >
> > > >> Currently it seems we have roughly as many test suite mechanisms as we
> > > >> have developers interested in using and supporting them.
> > > >>
> > > >> I imagine you ought to write an RFC proposal on your unittest approach
> > > >> before it is integrated into MapServer.
> > > >>
> > > >
> > > > If that is necessary I'll do it, but I'd better code than write.
> > >
> > > I'm sure we would all rather code than write RFCs, but we decided to
> > > require RFCs for a certain class of changes in MapServer to ensure
> > > we have coherent progress, avoid bloat, and fine tune approaches to
> > > new features before they we are committed to them.
> > >
> > > You can certainly maintain a test harnass on your own, but if you
> > > want it in the tree I think it deserves an RFC.  As per ms-rfc-1,
> > > Steve is the final arbiter of what does or does not need an RFC.
> > >
> > > Please don't consider my feedback as saying I don't want C unit
> > > tests.  If there is a good case for why we need them instead of
> > > python unit tests, and if you are willing to do the work to establish
> > > them then I'm keen on it.
> > >
> > > Best regards,
> > > --
> > > ---------------------------------------+--------------------------------------
> > > I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
> > > light and sound - activate the windows | http://pobox.com/~warmerdam
> > > and watch the world go round - Rush    | Geospatial Programmer for Rent
> > >
> > >
> >
>