[Mapserver-dev] Performance loss between 3.5 and 4.2

[Ed]

Daniel -

That's fine, but this discussion was started because of a user question <g>!

It's relatively easy for developers to create tests to check expected changes in code they're working on, and that's certainly a good idea.  The toughest part is checking for "I pushed a button over here and performance got worse over there" problems, but certainly a first-line defense is a very good start.

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
ed at topozone.com
(978) 251-4242  

-----Original Message-----
From: Daniel Morissette [mailto:dmorissette at dmsolutions.ca] 
Sent: Friday, September 24, 2004 4:19 PM
To: Ed McNierney
Cc: Sean Gillies;mapserver-dev at lists.gis.umn.edu
Subject: Re: [Mapserver-dev] Performance loss between 3.5 and 4.2

Ed McNierney wrote:
> Sean -
> 
> While I don't object to the goal, the chief difficulty is that MapServer covers such a very wide range of capabilities, with a very wide range of non-MapServer supporting libraries, that it's very difficult (IMHO) to create a test that is either comprehensive or useful.  It may be useful to have test suites to look for (a) validation of alleged performance-improving fixes or (b) changes that inadvertently cause a serious performance problem.  But the value of those suites is limited to keeping developers on their toes, not helping end users understand whether they should expect MapServer X to perform differently than MapServer Y for THEIR application - which is usually the only one they care about <g>.
> 

We're on the -dev list here, not on the -users list.  ;)  I agree that benchmarks don't mean much to users since the performance depends on too many factors. I think what Sean is talking about is a tool for
developers: tests for typical scenarios, combined with timing measurements to automatically find out if things were broken or performance was made worse by a given change to the code. Over time as we add more tests to this suite we may end up being able to cover all aspects of the software.

Daniel
-- 
------------------------------------------------------------
  Daniel Morissette               dmorissette at dmsolutions.ca
  DM Solutions Group              http://www.dmsolutions.ca/
------------------------------------------------------------