Issue #1664: Unit Test support for mapserver

Thu Feb 23 06:15:50 EST 2006

Hi Clint.

The article you are quoting was written in 2000, 6 years ago! I am
sure that in 2006 you will surely find a skille developer willing to
write unit tests (at least in the Java arena).

As of QA testing: I am not saying it is not necessary or usefule, but
mapserver could'n afford it anyway. The only thing it can afford is to
have the developers writing/running the tests themselves (as it is now
anyway).

Umberto

On 2/18/06, Clint Johnson <cjohnson at refractions.net> wrote:
> I agree with Ned on this issue.
>
> Sure Unit Testing can be useful; however, there is a *big* between unit
> testing and test driven development; that RFC depicts test driven
> development, without actually saying the name "test driven development."
>
> Unit testing is in essence a procedure to validate a particular piece of
> code ... test driven development on the other hand is a methodology in
> which tests/contracts are developed apriori.
>
> Unit testing is useful in testing core functionality, and it is only
> valuable insofar as it helps a development project stay on track in
> terms of costing less time/money/effort with the unit tests than without
> them.  However to test EVERYTHING incurs a significant chuck of time and
> money (as every test itself must be updated and maintained).
>
> Joel's blog talks about testing in the broadest sense:
> - [Top five wrong reasons you don't have testers]
> http://www.joelonsoftware.com/articles/fog0000000067.html
> - [Near the bottom, "Back to Tamir"]
> http://www.joelonsoftware.com/items/2004/12/06.html
>
> Personally, I see imposing the test driven development methodology upon
> any project will drive talented developers, such as Ned, away.
> Can test driven development replace a skilled tester? No.
> Can test driven development replace a poor design? No.
> Can test driven development stop bugs from completely appearing? No,
> "experienced developer is about 100 times less likely to write bugs that
> will be uncovered during unit tests than a beginner"
> Is it cheaper for a tester to test rather than a developer to test? Yes.
>
> Unit testing is handy; however, taken at an extreme, test driven
> development can go hand-in-hand with the law of diminishing returns.
>
> Just my 2 cents...
>
> Clint
>
>
>
>
>
> Ned Harding wrote:
> > Not that I have a vote, but my company is a infrequent but regular
> > contributor to MapServer...
> >
> > If the bar for contributions is raised so high we would be more likely
> > to fork MapServer and move on than bother with the bureaucracy...  My
> > experience is an initiative like this would only work if we (the
> > MapServer community) had a full time developer to maintain the test
> > suite.  In larger companies that might be reasonable, but in a volunteer
> > driven project like MapServer I see it driving developers away.
> >
> > I have found (and fixed) my share of crash bugs in MapServer and maybe
> > even a small regression bug or 2, but overall the quality of the code
> > has been acceptable.  I would be MUCH more comfortable waiting until a
> > stable outward C API is complete and then doing a functional test suite
> > because I would see that have 80% of the benefit with 20% of the work.
> > As Frank pointed out, so many of the issues can only be tested by a
> > human looking at an image, a few images with attached instructions for
> > how to evaluate quality in several areas would go a long way.
> >
> > I have enjoyed working with all the regular contributors here and have
> > never been afraid that by embedding MapServer in our products the
> > quality would suffer.  This feels like a solution in search of a
> > problem.
> >
> > Just my 2 cents...
> >
> > Ned.
> >
> > P.S.  If we are really going to change the way things are done in
> > MapServer, I think a switch to C++ would make more of a difference for
> > long term code quality...  (Let the flame war begin!)
> >
> > -----Original Message-----
> > From: UMN MapServer Developers List [mailto:MAPSERVER-DEV at LISTS.UMN.EDU]
> > On Behalf Of Umberto Nicoletti
> > Sent: Friday, February 17, 2006 3:05 AM
> > To: MAPSERVER-DEV at LISTS.UMN.EDU
> > Subject: Re: [UMN_MAPSERVER-DEV] Issue #1664: Unit Test support for
> > mapserver
> >
> > I have published a first public draft of rfc-12: Unit tests
> >
> > http://mapserver.gis.umn.edu/development/rfc/ms-rfc-12
> >
> > Please share your thoughts.
> >
> > Regards,
> > Umberto
> >
> >
> > On 2/16/06, Umberto Nicoletti <umberto.nicoletti at gmail.com> wrote:
> >
> >> I am trying to add an rfc, could someone enable rfc editing on the
> >> plone site for my account (unicoletti) or explain how to write one?
> >>
> >> Regards,
> >> Umberto
> >>
> >> On 2/15/06, Umberto Nicoletti <umberto.nicoletti at gmail.com> wrote:
> >>
> >>> Frank,
> >>> first of all thanks for your comments.
> >>>
> >>> On 2/14/06, Frank Warmerdam <warmerdam at pobox.com> wrote:
> >>>
> >>>> Umberto Nicoletti wrote:
> >>>>
> >>>>> 'Course I am, as a matter of facts your tests are functional
> >>>>> tests (a rather different beast from unit tests), while Sean's
> >>>>> are unit tests, but for python Mapscript.
> >>>>>
> >>>> Umberto,
> >>>>
> >>>> OK, cool.
> >>>>
> >>>>
> >>>>> The unit tests I am introducing can go deeper and test the
> >>>>> intimate of mapserver's code. Also unit tests can test a very
> >>>>> limited and well defined subset of code, making memory
> >>>>> leaks/memory misbehaviours/threading issue easier to spot. For
> >>>>> instance I have written unit tests for the connection pooling
> >>>>> which is not covered by Sean's tests because connection pooling
> >>>>> functions are not exposed by mapscript and neither by yours.
> >>>>>
> >>>> I think running valgrind on shp2img or mapserv sessions can
> >>>> identify most memory leaks.  You are quite right that none of the
> >>>> existing tests address pooling, or threading issues and it may be
> >>>> challenging to do this at any level other than the C API (as you
> >>>> suggest).
> >>>>
> >>>>
> >>>>> IMHO it is not the case I have to argue in favour of unit tests,
> >>>>>
> >
> >
> >>>>> but quite the contrary, as the beneficial effects of unit
> >>>>> testing seem pretty obvious to me (and to anybody who's ever
> >>>>>
> > used them).
> >
> >>>> I'm not arguing that there is no benefit to unit tests, and I have
> >>>>
> >
> >
> >>>> used them.  But if C level unit tests are to reach anything near
> >>>> their full value, it is necessary to get others involved in
> >>>> extending them and running them.
> >>>>
> >>> Agreed.
> >>>
> >>> IMO Unit testing is not an opt-in, but a programming approach that
> >>> must be used by all developers, that is a requirement.
> >>> This is a clear cut case (to quote Tarantino's Jackie Brown) in my
> >>> mind: either unit tests get into the repository AND EVERYBODY adopts
> >>>
> >
> >
> >>> them or they don't get into the repository at all.
> >>>
> >>>
> >>>>> The reasons are:
> >>>>> - better code
> >>>>> - safer modifications
> >>>>> - more confidence
> >>>>> - regression testing
> >>>>> - automation of repetitive tasks
> >>>>> - integration in the build process
> >>>>>
> >>> I forgot to add the documentation value that unit tests add to
> >>>
> > existing code.
> >
> >>>> I don't see that the above items require another testing mechanism
> >>>>
> >
> >
> >>>> with the possible exception of things that cannot practically be
> >>>> unittested via the python mapscript tests, or functionally tested
> >>>> via msautotest.
> >>>>
> >>>>
> >>> Unit tests do not replace functional testing, rather functional
> >>> tests complete unit tests.
> >>>
> >>>
> >>>>> See for instance bug 1629: if we had unit tests we could have
> >>>>> committed the modifications to 4.8.1 already without having to
> >>>>> recall why in the first we made those changes in the first
> >>>>>
> > place.
> >
> >>>> I have briefly reviewed Bug 1629 and I cannot see why a python
> >>>> mapscript or msautotest test item would have been inadequate to
> >>>> handle it.  Of course, there isn't any such test for PostGIS
> >>>> support in mapserver.  But we don't need a new approach, we just
> >>>> need to more fully fill out the existing approach.
> >>>>
> >>>>
> >>> Of course you can do it with mapscript or msautotest, but that
> >>> requires a postgis setup which not everybody has available.
> >>> With C unit tests you can just test the query composer function and
> >>> verify that the string it produces is the one you are expecting,
> >>> dropping entirely the need for a full fledged build environment.
> >>> Of course this kind of testing requires refactoring of existing code
> >>>
> >
> >
> >>> to make it more easily testable.
> >>> For an example see the two methods I have added to the connection
> >>>
> > pooling apis:
> >
> >>> int  msConnPoolGetConnectionCount(); int
> >>> msConnPoolGetConnectionMax();
> >>>
> >>> Best regards,
> >>> Umberto
> >>>
> >>>
> >>>>>> Currently it seems we have roughly as many test suite
> >>>>>> mechanisms as we have developers interested in using and
> >>>>>>
> > supporting them.
> >
> >>>>>> I imagine you ought to write an RFC proposal on your unittest
> >>>>>> approach before it is integrated into MapServer.
> >>>>>>
> >>>>>>
> >>>>> If that is necessary I'll do it, but I'd better code than write.
> >>>>>
> >>>> I'm sure we would all rather code than write RFCs, but we decided
> >>>> to require RFCs for a certain class of changes in MapServer to
> >>>> ensure we have coherent progress, avoid bloat, and fine tune
> >>>> approaches to new features before they we are committed to them.
> >>>>
> >>>> You can certainly maintain a test harnass on your own, but if you
> >>>> want it in the tree I think it deserves an RFC.  As per ms-rfc-1,
> >>>> Steve is the final arbiter of what does or does not need an RFC.
> >>>>
> >>>> Please don't consider my feedback as saying I don't want C unit
> >>>> tests.  If there is a good case for why we need them instead of
> >>>> python unit tests, and if you are willing to do the work to
> >>>> establish them then I'm keen on it.
> >>>>
> >>>> Best regards,
> >>>> --
> >>>> ---------------------------------------+--------------------------
> >>>> ---------------------------------------+------------
> >>>> I set the clouds in motion - turn up   | Frank Warmerdam,
> >>>>
> > warmerdam at pobox.com
> >
> >>>> light and sound - activate the windows |
> >>>>
> > http://pobox.com/~warmerdam
> >
> >>>> and watch the world go round - Rush    | Geospatial Programmer for
> >>>>
> > Rent
> >
> >>>>
>
>
> --
> Clint Johnson
> Refractions Research Inc.
> Cjohnson at refractions.net
>