Issue #1664: Unit Test support for mapserver

Thu Feb 23 06:13:45 EST 2006

Hi Ned,
Thanks for writing. Comments inline.

On 2/17/06, Ned Harding <nharding at extendthereach.com> wrote:
> Not that I have a vote, but my company is a infrequent but regular
> contributor to MapServer...
>
> If the bar for contributions is raised so high we would be more likely
> to fork MapServer and move on than bother with the bureaucracy...

I say it again: any serious developer would be doing testing anyway,
so why not doing it in a way
that it can be automatically run at every build instead of only once?

> My
> experience is an initiative like this would only work if we (the
> MapServer community) had a full time developer to maintain the test
> suite.  In larger companies that might be reasonable, but in a volunteer
> driven project like MapServer I see it driving developers away.

Then what about all those Perl modules (mentioned in my answer to
Frank) and the many Java Open source programs that do have unit tests?

>
> I have found (and fixed) my share of crash bugs in MapServer and maybe
> even a small regression bug or 2, but overall the quality of the code
> has been acceptable.  I would be MUCH more comfortable waiting until a
> stable outward C API is complete and then doing a functional test suite
> because I would see that have 80% of the benefit with 20% of the work.
> As Frank pointed out, so many of the issues can only be tested by a
> human looking at an image, a few images with attached instructions for
> how to evaluate quality in several areas would go a long way.
>

But mapserver is not only about displaying images: for instance XML in
WFS queries, sql strings in postgis, just to name a few. The latter
should be more readily testable.
As of images we could look at what others have done: the GD module for
perl and the msautotest suite.

Best Regards,
Umberto

> I have enjoyed working with all the regular contributors here and have
> never been afraid that by embedding MapServer in our products the
> quality would suffer.  This feels like a solution in search of a
> problem.
>
> Just my 2 cents...
>
> Ned.
>
> P.S.  If we are really going to change the way things are done in
> MapServer, I think a switch to C++ would make more of a difference for
> long term code quality...  (Let the flame war begin!)
>
> -----Original Message-----
> From: UMN MapServer Developers List [mailto:MAPSERVER-DEV at LISTS.UMN.EDU]
> On Behalf Of Umberto Nicoletti
> Sent: Friday, February 17, 2006 3:05 AM
> To: MAPSERVER-DEV at LISTS.UMN.EDU
> Subject: Re: [UMN_MAPSERVER-DEV] Issue #1664: Unit Test support for
> mapserver
>
> I have published a first public draft of rfc-12: Unit tests
>
> http://mapserver.gis.umn.edu/development/rfc/ms-rfc-12
>
> Please share your thoughts.
>
> Regards,
> Umberto
>
>
> On 2/16/06, Umberto Nicoletti <umberto.nicoletti at gmail.com> wrote:
> > I am trying to add an rfc, could someone enable rfc editing on the
> > plone site for my account (unicoletti) or explain how to write one?
> >
> > Regards,
> > Umberto
> >
> > On 2/15/06, Umberto Nicoletti <umberto.nicoletti at gmail.com> wrote:
> > > Frank,
> > > first of all thanks for your comments.
> > >
> > > On 2/14/06, Frank Warmerdam <warmerdam at pobox.com> wrote:
> > > > Umberto Nicoletti wrote:
> > > > > 'Course I am, as a matter of facts your tests are functional
> > > > > tests (a rather different beast from unit tests), while Sean's
> > > > > are unit tests, but for python Mapscript.
> > > >
> > > > Umberto,
> > > >
> > > > OK, cool.
> > > >
> > > > > The unit tests I am introducing can go deeper and test the
> > > > > intimate of mapserver's code. Also unit tests can test a very
> > > > > limited and well defined subset of code, making memory
> > > > > leaks/memory misbehaviours/threading issue easier to spot. For
> > > > > instance I have written unit tests for the connection pooling
> > > > > which is not covered by Sean's tests because connection pooling
> > > > > functions are not exposed by mapscript and neither by yours.
> > > >
> > > > I think running valgrind on shp2img or mapserv sessions can
> > > > identify most memory leaks.  You are quite right that none of the
> > > > existing tests address pooling, or threading issues and it may be
> > > > challenging to do this at any level other than the C API (as you
> > > > suggest).
> > > >
> > > > > IMHO it is not the case I have to argue in favour of unit tests,
>
> > > > > but quite the contrary, as the beneficial effects of unit
> > > > > testing seem pretty obvious to me (and to anybody who's ever
> used them).
> > > >
> > > > I'm not arguing that there is no benefit to unit tests, and I have
>
> > > > used them.  But if C level unit tests are to reach anything near
> > > > their full value, it is necessary to get others involved in
> > > > extending them and running them.
> > >
> > > Agreed.
> > >
> > > IMO Unit testing is not an opt-in, but a programming approach that
> > > must be used by all developers, that is a requirement.
> > > This is a clear cut case (to quote Tarantino's Jackie Brown) in my
> > > mind: either unit tests get into the repository AND EVERYBODY adopts
>
> > > them or they don't get into the repository at all.
> > >
> > > >
> > > > > The reasons are:
> > > > > - better code
> > > > > - safer modifications
> > > > > - more confidence
> > > > > - regression testing
> > > > > - automation of repetitive tasks
> > > > > - integration in the build process
> > > >
> > >
> > > I forgot to add the documentation value that unit tests add to
> existing code.
> > >
> > > > I don't see that the above items require another testing mechanism
>
> > > > with the possible exception of things that cannot practically be
> > > > unittested via the python mapscript tests, or functionally tested
> > > > via msautotest.
> > > >
> > >
> > > Unit tests do not replace functional testing, rather functional
> > > tests complete unit tests.
> > >
> > > > > See for instance bug 1629: if we had unit tests we could have
> > > > > committed the modifications to 4.8.1 already without having to
> > > > > recall why in the first we made those changes in the first
> place.
> > > >
> > > > I have briefly reviewed Bug 1629 and I cannot see why a python
> > > > mapscript or msautotest test item would have been inadequate to
> > > > handle it.  Of course, there isn't any such test for PostGIS
> > > > support in mapserver.  But we don't need a new approach, we just
> > > > need to more fully fill out the existing approach.
> > > >
> > >
> > > Of course you can do it with mapscript or msautotest, but that
> > > requires a postgis setup which not everybody has available.
> > > With C unit tests you can just test the query composer function and
> > > verify that the string it produces is the one you are expecting,
> > > dropping entirely the need for a full fledged build environment.
> > > Of course this kind of testing requires refactoring of existing code
>
> > > to make it more easily testable.
> > > For an example see the two methods I have added to the connection
> pooling apis:
> > >
> > > int  msConnPoolGetConnectionCount(); int
> > > msConnPoolGetConnectionMax();
> > >
> > > Best regards,
> > > Umberto
> > >
> > > > >> Currently it seems we have roughly as many test suite
> > > > >> mechanisms as we have developers interested in using and
> supporting them.
> > > > >>
> > > > >> I imagine you ought to write an RFC proposal on your unittest
> > > > >> approach before it is integrated into MapServer.
> > > > >>
> > > > >
> > > > > If that is necessary I'll do it, but I'd better code than write.
> > > >
> > > > I'm sure we would all rather code than write RFCs, but we decided
> > > > to require RFCs for a certain class of changes in MapServer to
> > > > ensure we have coherent progress, avoid bloat, and fine tune
> > > > approaches to new features before they we are committed to them.
> > > >
> > > > You can certainly maintain a test harnass on your own, but if you
> > > > want it in the tree I think it deserves an RFC.  As per ms-rfc-1,
> > > > Steve is the final arbiter of what does or does not need an RFC.
> > > >
> > > > Please don't consider my feedback as saying I don't want C unit
> > > > tests.  If there is a good case for why we need them instead of
> > > > python unit tests, and if you are willing to do the work to
> > > > establish them then I'm keen on it.
> > > >
> > > > Best regards,
> > > > --
> > > > ---------------------------------------+--------------------------
> > > > ---------------------------------------+------------
> > > > I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> > > > light and sound - activate the windows |
> http://pobox.com/~warmerdam
> > > > and watch the world go round - Rush    | Geospatial Programmer for
> Rent
> > > >
> > > >
> > >
> >
>