Issue #1664: Unit Test support for mapserver

[Ned Harding]

Not that I have a vote, but my company is a infrequent but regular
contributor to MapServer...

If the bar for contributions is raised so high we would be more likely
to fork MapServer and move on than bother with the bureaucracy...  My
experience is an initiative like this would only work if we (the
MapServer community) had a full time developer to maintain the test
suite.  In larger companies that might be reasonable, but in a volunteer
driven project like MapServer I see it driving developers away.

I have found (and fixed) my share of crash bugs in MapServer and maybe
even a small regression bug or 2, but overall the quality of the code
has been acceptable.  I would be MUCH more comfortable waiting until a
stable outward C API is complete and then doing a functional test suite
because I would see that have 80% of the benefit with 20% of the work.
As Frank pointed out, so many of the issues can only be tested by a
human looking at an image, a few images with attached instructions for
how to evaluate quality in several areas would go a long way.

I have enjoyed working with all the regular contributors here and have
never been afraid that by embedding MapServer in our products the
quality would suffer.  This feels like a solution in search of a
problem.

Just my 2 cents...

Ned.

P.S.  If we are really going to change the way things are done in
MapServer, I think a switch to C++ would make more of a difference for
long term code quality...  (Let the flame war begin!)

-----Original Message-----
From: UMN MapServer Developers List [mailto:MAPSERVER-DEV at LISTS.UMN.EDU]
On Behalf Of Umberto Nicoletti
Sent: Friday, February 17, 2006 3:05 AM
To: MAPSERVER-DEV at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-DEV] Issue #1664: Unit Test support for
mapserver

I have published a first public draft of rfc-12: Unit tests

http://mapserver.gis.umn.edu/development/rfc/ms-rfc-12

Please share your thoughts.

Regards,
Umberto


On 2/16/06, Umberto Nicoletti <umberto.nicoletti at gmail.com> wrote:
> I am trying to add an rfc, could someone enable rfc editing on the 
> plone site for my account (unicoletti) or explain how to write one?
>
> Regards,
> Umberto
>
> On 2/15/06, Umberto Nicoletti <umberto.nicoletti at gmail.com> wrote:
> > Frank,
> > first of all thanks for your comments.
> >
> > On 2/14/06, Frank Warmerdam <warmerdam at pobox.com> wrote:
> > > Umberto Nicoletti wrote:
> > > > 'Course I am, as a matter of facts your tests are functional 
> > > > tests (a rather different beast from unit tests), while Sean's 
> > > > are unit tests, but for python Mapscript.
> > >
> > > Umberto,
> > >
> > > OK, cool.
> > >
> > > > The unit tests I am introducing can go deeper and test the 
> > > > intimate of mapserver's code. Also unit tests can test a very 
> > > > limited and well defined subset of code, making memory 
> > > > leaks/memory misbehaviours/threading issue easier to spot. For 
> > > > instance I have written unit tests for the connection pooling 
> > > > which is not covered by Sean's tests because connection pooling 
> > > > functions are not exposed by mapscript and neither by yours.
> > >
> > > I think running valgrind on shp2img or mapserv sessions can 
> > > identify most memory leaks.  You are quite right that none of the 
> > > existing tests address pooling, or threading issues and it may be 
> > > challenging to do this at any level other than the C API (as you 
> > > suggest).
> > >
> > > > IMHO it is not the case I have to argue in favour of unit tests,

> > > > but quite the contrary, as the beneficial effects of unit 
> > > > testing seem pretty obvious to me (and to anybody who's ever
used them).
> > >
> > > I'm not arguing that there is no benefit to unit tests, and I have

> > > used them.  But if C level unit tests are to reach anything near 
> > > their full value, it is necessary to get others involved in 
> > > extending them and running them.
> >
> > Agreed.
> >
> > IMO Unit testing is not an opt-in, but a programming approach that 
> > must be used by all developers, that is a requirement.
> > This is a clear cut case (to quote Tarantino's Jackie Brown) in my
> > mind: either unit tests get into the repository AND EVERYBODY adopts

> > them or they don't get into the repository at all.
> >
> > >
> > > > The reasons are:
> > > > - better code
> > > > - safer modifications
> > > > - more confidence
> > > > - regression testing
> > > > - automation of repetitive tasks
> > > > - integration in the build process
> > >
> >
> > I forgot to add the documentation value that unit tests add to
existing code.
> >
> > > I don't see that the above items require another testing mechanism

> > > with the possible exception of things that cannot practically be 
> > > unittested via the python mapscript tests, or functionally tested 
> > > via msautotest.
> > >
> >
> > Unit tests do not replace functional testing, rather functional 
> > tests complete unit tests.
> >
> > > > See for instance bug 1629: if we had unit tests we could have 
> > > > committed the modifications to 4.8.1 already without having to 
> > > > recall why in the first we made those changes in the first
place.
> > >
> > > I have briefly reviewed Bug 1629 and I cannot see why a python 
> > > mapscript or msautotest test item would have been inadequate to 
> > > handle it.  Of course, there isn't any such test for PostGIS 
> > > support in mapserver.  But we don't need a new approach, we just 
> > > need to more fully fill out the existing approach.
> > >
> >
> > Of course you can do it with mapscript or msautotest, but that 
> > requires a postgis setup which not everybody has available.
> > With C unit tests you can just test the query composer function and 
> > verify that the string it produces is the one you are expecting, 
> > dropping entirely the need for a full fledged build environment.
> > Of course this kind of testing requires refactoring of existing code

> > to make it more easily testable.
> > For an example see the two methods I have added to the connection
pooling apis:
> >
> > int  msConnPoolGetConnectionCount(); int  
> > msConnPoolGetConnectionMax();
> >
> > Best regards,
> > Umberto
> >
> > > >> Currently it seems we have roughly as many test suite 
> > > >> mechanisms as we have developers interested in using and
supporting them.
> > > >>
> > > >> I imagine you ought to write an RFC proposal on your unittest 
> > > >> approach before it is integrated into MapServer.
> > > >>
> > > >
> > > > If that is necessary I'll do it, but I'd better code than write.
> > >
> > > I'm sure we would all rather code than write RFCs, but we decided 
> > > to require RFCs for a certain class of changes in MapServer to 
> > > ensure we have coherent progress, avoid bloat, and fine tune 
> > > approaches to new features before they we are committed to them.
> > >
> > > You can certainly maintain a test harnass on your own, but if you 
> > > want it in the tree I think it deserves an RFC.  As per ms-rfc-1, 
> > > Steve is the final arbiter of what does or does not need an RFC.
> > >
> > > Please don't consider my feedback as saying I don't want C unit 
> > > tests.  If there is a good case for why we need them instead of 
> > > python unit tests, and if you are willing to do the work to 
> > > establish them then I'm keen on it.
> > >
> > > Best regards,
> > > --
> > > ---------------------------------------+--------------------------
> > > ---------------------------------------+------------
> > > I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
> > > light and sound - activate the windows |
http://pobox.com/~warmerdam
> > > and watch the world go round - Rush    | Geospatial Programmer for
Rent
> > >
> > >
> >
>