Issue #1664: Unit Test support for mapserver
Clint Johnson
cjohnson at REFRACTIONS.NET
Fri Feb 17 19:44:56 EST 2006
I agree with Ned on this issue.
Sure Unit Testing can be useful; however, there is a *big* between unit
testing and test driven development; that RFC depicts test driven
development, without actually saying the name "test driven development."
Unit testing is in essence a procedure to validate a particular piece of
code ... test driven development on the other hand is a methodology in
which tests/contracts are developed apriori.
Unit testing is useful in testing core functionality, and it is only
valuable insofar as it helps a development project stay on track in
terms of costing less time/money/effort with the unit tests than without
them. However to test EVERYTHING incurs a significant chuck of time and
money (as every test itself must be updated and maintained).
Joel's blog talks about testing in the broadest sense:
- [Top five wrong reasons you don't have testers]
http://www.joelonsoftware.com/articles/fog0000000067.html
- [Near the bottom, "Back to Tamir"]
http://www.joelonsoftware.com/items/2004/12/06.html
Personally, I see imposing the test driven development methodology upon
any project will drive talented developers, such as Ned, away.
Can test driven development replace a skilled tester? No.
Can test driven development replace a poor design? No.
Can test driven development stop bugs from completely appearing? No,
"experienced developer is about 100 times less likely to write bugs that
will be uncovered during unit tests than a beginner"
Is it cheaper for a tester to test rather than a developer to test? Yes.
Unit testing is handy; however, taken at an extreme, test driven
development can go hand-in-hand with the law of diminishing returns.
Just my 2 cents...
Clint
Ned Harding wrote:
> Not that I have a vote, but my company is a infrequent but regular
> contributor to MapServer...
>
> If the bar for contributions is raised so high we would be more likely
> to fork MapServer and move on than bother with the bureaucracy... My
> experience is an initiative like this would only work if we (the
> MapServer community) had a full time developer to maintain the test
> suite. In larger companies that might be reasonable, but in a volunteer
> driven project like MapServer I see it driving developers away.
>
> I have found (and fixed) my share of crash bugs in MapServer and maybe
> even a small regression bug or 2, but overall the quality of the code
> has been acceptable. I would be MUCH more comfortable waiting until a
> stable outward C API is complete and then doing a functional test suite
> because I would see that have 80% of the benefit with 20% of the work.
> As Frank pointed out, so many of the issues can only be tested by a
> human looking at an image, a few images with attached instructions for
> how to evaluate quality in several areas would go a long way.
>
> I have enjoyed working with all the regular contributors here and have
> never been afraid that by embedding MapServer in our products the
> quality would suffer. This feels like a solution in search of a
> problem.
>
> Just my 2 cents...
>
> Ned.
>
> P.S. If we are really going to change the way things are done in
> MapServer, I think a switch to C++ would make more of a difference for
> long term code quality... (Let the flame war begin!)
>
> -----Original Message-----
> From: UMN MapServer Developers List [mailto:MAPSERVER-DEV at LISTS.UMN.EDU]
> On Behalf Of Umberto Nicoletti
> Sent: Friday, February 17, 2006 3:05 AM
> To: MAPSERVER-DEV at LISTS.UMN.EDU
> Subject: Re: [UMN_MAPSERVER-DEV] Issue #1664: Unit Test support for
> mapserver
>
> I have published a first public draft of rfc-12: Unit tests
>
> http://mapserver.gis.umn.edu/development/rfc/ms-rfc-12
>
> Please share your thoughts.
>
> Regards,
> Umberto
>
>
> On 2/16/06, Umberto Nicoletti <umberto.nicoletti at gmail.com> wrote:
>
>> I am trying to add an rfc, could someone enable rfc editing on the
>> plone site for my account (unicoletti) or explain how to write one?
>>
>> Regards,
>> Umberto
>>
>> On 2/15/06, Umberto Nicoletti <umberto.nicoletti at gmail.com> wrote:
>>
>>> Frank,
>>> first of all thanks for your comments.
>>>
>>> On 2/14/06, Frank Warmerdam <warmerdam at pobox.com> wrote:
>>>
>>>> Umberto Nicoletti wrote:
>>>>
>>>>> 'Course I am, as a matter of facts your tests are functional
>>>>> tests (a rather different beast from unit tests), while Sean's
>>>>> are unit tests, but for python Mapscript.
>>>>>
>>>> Umberto,
>>>>
>>>> OK, cool.
>>>>
>>>>
>>>>> The unit tests I am introducing can go deeper and test the
>>>>> intimate of mapserver's code. Also unit tests can test a very
>>>>> limited and well defined subset of code, making memory
>>>>> leaks/memory misbehaviours/threading issue easier to spot. For
>>>>> instance I have written unit tests for the connection pooling
>>>>> which is not covered by Sean's tests because connection pooling
>>>>> functions are not exposed by mapscript and neither by yours.
>>>>>
>>>> I think running valgrind on shp2img or mapserv sessions can
>>>> identify most memory leaks. You are quite right that none of the
>>>> existing tests address pooling, or threading issues and it may be
>>>> challenging to do this at any level other than the C API (as you
>>>> suggest).
>>>>
>>>>
>>>>> IMHO it is not the case I have to argue in favour of unit tests,
>>>>>
>
>
>>>>> but quite the contrary, as the beneficial effects of unit
>>>>> testing seem pretty obvious to me (and to anybody who's ever
>>>>>
> used them).
>
>>>> I'm not arguing that there is no benefit to unit tests, and I have
>>>>
>
>
>>>> used them. But if C level unit tests are to reach anything near
>>>> their full value, it is necessary to get others involved in
>>>> extending them and running them.
>>>>
>>> Agreed.
>>>
>>> IMO Unit testing is not an opt-in, but a programming approach that
>>> must be used by all developers, that is a requirement.
>>> This is a clear cut case (to quote Tarantino's Jackie Brown) in my
>>> mind: either unit tests get into the repository AND EVERYBODY adopts
>>>
>
>
>>> them or they don't get into the repository at all.
>>>
>>>
>>>>> The reasons are:
>>>>> - better code
>>>>> - safer modifications
>>>>> - more confidence
>>>>> - regression testing
>>>>> - automation of repetitive tasks
>>>>> - integration in the build process
>>>>>
>>> I forgot to add the documentation value that unit tests add to
>>>
> existing code.
>
>>>> I don't see that the above items require another testing mechanism
>>>>
>
>
>>>> with the possible exception of things that cannot practically be
>>>> unittested via the python mapscript tests, or functionally tested
>>>> via msautotest.
>>>>
>>>>
>>> Unit tests do not replace functional testing, rather functional
>>> tests complete unit tests.
>>>
>>>
>>>>> See for instance bug 1629: if we had unit tests we could have
>>>>> committed the modifications to 4.8.1 already without having to
>>>>> recall why in the first we made those changes in the first
>>>>>
> place.
>
>>>> I have briefly reviewed Bug 1629 and I cannot see why a python
>>>> mapscript or msautotest test item would have been inadequate to
>>>> handle it. Of course, there isn't any such test for PostGIS
>>>> support in mapserver. But we don't need a new approach, we just
>>>> need to more fully fill out the existing approach.
>>>>
>>>>
>>> Of course you can do it with mapscript or msautotest, but that
>>> requires a postgis setup which not everybody has available.
>>> With C unit tests you can just test the query composer function and
>>> verify that the string it produces is the one you are expecting,
>>> dropping entirely the need for a full fledged build environment.
>>> Of course this kind of testing requires refactoring of existing code
>>>
>
>
>>> to make it more easily testable.
>>> For an example see the two methods I have added to the connection
>>>
> pooling apis:
>
>>> int msConnPoolGetConnectionCount(); int
>>> msConnPoolGetConnectionMax();
>>>
>>> Best regards,
>>> Umberto
>>>
>>>
>>>>>> Currently it seems we have roughly as many test suite
>>>>>> mechanisms as we have developers interested in using and
>>>>>>
> supporting them.
>
>>>>>> I imagine you ought to write an RFC proposal on your unittest
>>>>>> approach before it is integrated into MapServer.
>>>>>>
>>>>>>
>>>>> If that is necessary I'll do it, but I'd better code than write.
>>>>>
>>>> I'm sure we would all rather code than write RFCs, but we decided
>>>> to require RFCs for a certain class of changes in MapServer to
>>>> ensure we have coherent progress, avoid bloat, and fine tune
>>>> approaches to new features before they we are committed to them.
>>>>
>>>> You can certainly maintain a test harnass on your own, but if you
>>>> want it in the tree I think it deserves an RFC. As per ms-rfc-1,
>>>> Steve is the final arbiter of what does or does not need an RFC.
>>>>
>>>> Please don't consider my feedback as saying I don't want C unit
>>>> tests. If there is a good case for why we need them instead of
>>>> python unit tests, and if you are willing to do the work to
>>>> establish them then I'm keen on it.
>>>>
>>>> Best regards,
>>>> --
>>>> ---------------------------------------+--------------------------
>>>> ---------------------------------------+------------
>>>> I set the clouds in motion - turn up | Frank Warmerdam,
>>>>
> warmerdam at pobox.com
>
>>>> light and sound - activate the windows |
>>>>
> http://pobox.com/~warmerdam
>
>>>> and watch the world go round - Rush | Geospatial Programmer for
>>>>
> Rent
>
>>>>
--
Clint Johnson
Refractions Research Inc.
Cjohnson at refractions.net
More information about the mapserver-dev
mailing list