Benchmarks

[Arnulf Christl]

OK, 2 more cent from me:

Its not very helpful to know that one set of polygons will take x% more
or less on one software or another. Most everybody needs to put labels
on the geometry (some with conflict avoidance), classify it, create
strange queries in the DB or compute results in views (where most of the
performance bottleneck sits in the database). If you use cascading
architectures (either as overlay in a thin client or collection in a
WMS) and sort your data in an intelligent way, performance can grow
enormously. Think about parallelling four small machines for load
distribution (a lot more economical than one big xeon multiprocessor) -
you can't do that with a closed license system - therfore you wouldn't
be doing it in a benchmark test environment. Even the way that data is
fragmented on the hardware can make a difference.

And it goes on and on like that. I'm not saying that its impossible to
benchmark mapserver software, but what good are the results going to be
for our day to day use?

Besides, with a general performance measured in seconds for each request
we (GIS folks) are the underdogs when it comes to web performance
anyway. Everytime any map server gets /. it will just cease to exist
within minutes, no matter what software or how powerful a system it is
running on - at least thats what I have observed. Don't ever get your
server  too well known...

Last but not least - if any major software seller would have the heart
to make a performance benchmark against UMN MapServer, wouldn't you
think they'd have done so long ago?
:-)

Cheers, Arnulf.

Ed McNierney wrote:

>Jacob -
>
>Things like the amount of RAM and the physical disk hardware can make a big difference in performance, so a different machine may not produce "slightly" different results.
>
>I'm not saying that benchmarks aren't reproducible.  I'm saying that they are so dependent on the complete system hardware/software/data/application configuration that it's very hard to compare DIFFERENT systems.  Knowing that MapServer is faster or slower than Product X by 10% on a very specific configuration tells me very little about how it will perform on MY hardware and MY configuration, which is usually what users are interested in when using benchmark tests.
>
>    - Ed
>Ed McNierney
>President and Chief Mapmaker
>TopoZone.com / Maps a la carte, Inc.
>73 Princeton Street, Suite 305
>North Chelmsford, MA  01863
>Phone: +1 978 251-4242   Fax: +1 978 251-1396
>
>
>________________________________
>
>From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Jacob Delfos
>Sent: Tuesday, August 17, 2004 7:30 PM
>To: MAPSERVER-USERS at LISTS.UMN.EDU
>Subject: Re: [UMN_MAPSERVER-USERS] Benchmarks
>
>
>I think if you take 1 server with a generic setup, install both systems, load the same datasets (e.g. a roadnetwork only, or an image only), and load the webpage 20 times from a browser on that same server, or a sole machine connected to the server (crossover), you'd get a pretty good idea, isn't it? You'd be able to say the same dataset with the same extents took x times longer on system y. A different machine may produce slightly different results, depending on different bottlenecks, but I'd say it would be reasonably similar.
>
>-----Original Message-----
>From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Arnulf Christl
>Sent: 17 August 2004 15:55
>To: MAPSERVER-USERS at LISTS.UMN.EDU
>Subject: Re: [UMN_MAPSERVER-USERS] Benchmarks
>
>
>Hi,
>the questions of benchmarking comes up every now an then. There has been
>quite a lot of additional communication going on, some of which the
>posters decided to keep off the list because of all the implications Ed
>implicated...
>:-)
>Maybe we should gather some postings covering aspects why it is
>pracitcally impossible to produce meaningful and verifiable benchmarks.
>I remember that somebody wanted to do that but lost track what came out
>of it. Does anybody know or is interested?
>
>Basically it burns down to the problem that geodata and requirements are
>too heterogeneous to be comparable so that you can always compare only
>one dataset with one specific question in mind (the complexity issue).
>
>Arnulf.
>
>Ed McNierney wrote:
>
>
>
>>Ken -
>>
>>I think the complexity is one issue.  I think the other issue is that
>>these are expensive products, and it's hard to see an independent lab
>>doing the tests unless the vendors donate their software.  And if ESRI
>>is unwilling to publish the results of the tests they control, I'd doubt
>>very much that they'd be willing to give free software so someone else
>>could publish tests they DON'T control!
>>
>>       - Ed
>>
>>-----Original Message-----
>>From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
>>Behalf Of Ken Waters
>>Sent: Monday, August 16, 2004 2:33 PM
>>To: MAPSERVER-USERS at LISTS.UMN.EDU
>>Subject: Re: [UMN_MAPSERVER-USERS] Benchmarks
>>
>>All,
>>
>>Last weekend I was at an ESRI seminar, in conjunction with their annual
>>conference.  Somehow this question came up.  The ESRI "unofficial"
>>answer was that they have such statistics but have elected to not share
>>them.  Fair enough.
>>
>>I think it would be great for someone who is not associated with any
>>company or other interests to do an objective test of the different
>>solutions.
>>Maybe this hasn't happened due to the complexity that Ed refers to here.
>>
>>Ken
>>
>>Ed McNierney wrote:
>>
>>
>>
>>
>>
>>>Rich -
>>>
>>>Are you aware of any benchmarks that compare ArcIMS to anything?
>>>Benchmarking Internet map servers is a extremely complicated subject,
>>>and I have not seen anyone make a serious attempt at doing it for ANY
>>>products.
>>>
>>>       - Ed
>>>
>>>Ed McNierney
>>>President and Chief Mapmaker
>>>TopoZone.com / Maps a la carte, Inc.
>>>73 Princeton Street, Suite 305
>>>North Chelmsford, MA  01863
>>>ed at topozone.com
>>>(978) 251-4242
>>>
>>>-----Original Message-----
>>>From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]
>>>On Behalf Of Rich Binns
>>>Sent: Monday, August 16, 2004 1:52 PM
>>>To: MAPSERVER-USERS at LISTS.UMN.EDU
>>>Subject: [UMN_MAPSERVER-USERS] Benchmarks
>>>
>>>Was wondering if anyone could direct me to where I could find
>>>benchmark info comparing Mapserver to proprietary solutions,
>>>
>>>
>>>
>>>
>>specifically ArcIMS.
>>
>>
>>
>>
>>
>
>
>--
>----------------------------
>CCGIS Christl & Stamm GbR
>----------------------------
>http://www.ccgis.de
>http://www.mapbender.org
>----------------------------
>Service: +49 (0)228 90826 0
>Direct:  +49 (0)228 90826 23
>Fax:     +49 (0)228 90826 11
>----------------------------
>CCGIS Christl & Stamm GbR
>Siemensstraße 8
>53121 Bonn
>GERMANY
>
><mailto:jacob.delfos at maunsell.com>
>
>