[Mapserver-dev] Performance loss between 3.5 and 4.2

[Sean Gillies]

Moving your reply to the bottom ...

On Sep 24, 2004, at 1:52 PM, Ed McNierney wrote:

>> -----Original Message-----
>> From: mapserver-dev-admin at lists.gis.umn.edu 
>> [mailto:mapserver-dev-admin at lists.gis.umn.edu] On Behalf Of Sean 
>> Gillies
>> Sent: Friday, September 24, 2004 2:21 PM
>> To: Daniel Morissette
>> Cc: mapserver-dev at lists.gis.umn.edu
>> Subject: Re: [Mapserver-dev] Performance loss between 3.5 and 4.2
>>
>> On Sep 24, 2004, at 11:10 AM, Daniel Morissette wrote:
>>
>>> Sean Gillies wrote:
>>>> Until the mapserver project adopts some regularly run performance
>>>> benchmarks, all evidence that mapserver is slower or faster than it
>>>> used to be will be only anecdotal.  Anyone interested?
>>>
>>> I think we'd be interested, but what do you have in mind exactly?
>>>
>>
>> Nothing concrete, just ideas ...
>>
>> For starters, could be as simple as running shp2img every night for 
>> each CVS branch using the same test data (or as identical as 
>> possible).  One platform would be enough to start.  Write output to a 
>> file that is published through the web.  Maybe have a chart of 
>> performance versus calendar time with a line for each branch.  So 
>> when somebody sees a drastic dip in performance over a few nights, 
>> they can look in CVS to see what's been committed -- and see that 
>> I've been working on brush caching in mapgd.c (for example).  I don't 
>> think we'd necessarily even need to use a nightly profiler, just time 
>> the process as accurately and consistently as we can.
>>
>> That takes care of the future performance history.  For the past, we 
>> could travel back in time with CVS snapshots :).
>>
>> Sean
>>
> _______________________________________________
> Sean -
>
> While I don't object to the goal, the chief difficulty is that 
> MapServer covers such a very wide range of capabilities, with a very 
> wide range of non-MapServer supporting libraries, that it's very 
> difficult (IMHO) to create a test that is either comprehensive or 
> useful.  It may be useful to have test suites to look for (a) 
> validation of alleged performance-improving fixes or (b) changes that 
> inadvertently cause a serious performance problem.  But the value of 
> those suites is limited to keeping developers on their toes, not 
> helping end users understand whether they should expect MapServer X to 
> perform differently than MapServer Y for THEIR application - which is 
> usually the only one they care about <g>.
>
> 	- Ed
>

I think that benchmarking mapserver querying and drawing of shapefile
features would thinly cover about half of use cases.  Add tiled TIFF
rasters through GDAL and postgis vectors, and we're probably at 3/4
to 4/5 of use cases (estimating from Tyler Mitchell's surveys).

The benchmarks aren't intended for end users anyway, although it might
be helpful for someone to compare their own tests against the core
mapserver benchmarks.  If the mapserver core speeds up between releases
and the user experiences the opposite, the user can focus on the 
differences
in their application -- data drivers, data optimization, tiling,
compression, etc. -- and be reasonably hopeful that improving the data
side will speed up their app.

I think there are some gains to be found in the mapserver rendering 
core,
but without benchmarking it will be hard to measure them, or measure
loss of performance vs gained functionality, in a meaningful way.

Appreciate the comments, Ed!
cheers,
Sean

--
Sean Gillies
sgillies at frii dot com
http://users.frii.com/sgillies