Benchmarks

[Ed McNierney]

Jacob -
 
Things like the amount of RAM and the physical disk hardware can make a big difference in performance, so a different machine may not produce "slightly" different results.
 
I'm not saying that benchmarks aren't reproducible.  I'm saying that they are so dependent on the complete system hardware/software/data/application configuration that it's very hard to compare DIFFERENT systems.  Knowing that MapServer is faster or slower than Product X by 10% on a very specific configuration tells me very little about how it will perform on MY hardware and MY configuration, which is usually what users are interested in when using benchmark tests.
 
    - Ed
Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: +1 978 251-4242   Fax: +1 978 251-1396 


________________________________

From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Jacob Delfos
Sent: Tuesday, August 17, 2004 7:30 PM
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] Benchmarks


I think if you take 1 server with a generic setup, install both systems, load the same datasets (e.g. a roadnetwork only, or an image only), and load the webpage 20 times from a browser on that same server, or a sole machine connected to the server (crossover), you'd get a pretty good idea, isn't it? You'd be able to say the same dataset with the same extents took x times longer on system y. A different machine may produce slightly different results, depending on different bottlenecks, but I'd say it would be reasonably similar.
 
-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of Arnulf Christl
Sent: 17 August 2004 15:55
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] Benchmarks
 

Hi,
the questions of benchmarking comes up every now an then. There has been
quite a lot of additional communication going on, some of which the
posters decided to keep off the list because of all the implications Ed
implicated...
:-)
Maybe we should gather some postings covering aspects why it is
pracitcally impossible to produce meaningful and verifiable benchmarks.
I remember that somebody wanted to do that but lost track what came out
of it. Does anybody know or is interested?
 
Basically it burns down to the problem that geodata and requirements are
too heterogeneous to be comparable so that you can always compare only
one dataset with one specific question in mind (the complexity issue).
 
Arnulf.
 
Ed McNierney wrote:
 
>Ken -
>
>I think the complexity is one issue.  I think the other issue is that
>these are expensive products, and it's hard to see an independent lab
>doing the tests unless the vendors donate their software.  And if ESRI
>is unwilling to publish the results of the tests they control, I'd doubt
>very much that they'd be willing to give free software so someone else
>could publish tests they DON'T control!
>
>        - Ed
>
>-----Original Message-----
>From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
>Behalf Of Ken Waters
>Sent: Monday, August 16, 2004 2:33 PM
>To: MAPSERVER-USERS at LISTS.UMN.EDU
>Subject: Re: [UMN_MAPSERVER-USERS] Benchmarks
>
>All,
>
>Last weekend I was at an ESRI seminar, in conjunction with their annual
>conference.  Somehow this question came up.  The ESRI "unofficial"
>answer was that they have such statistics but have elected to not share
>them.  Fair enough.
>
>I think it would be great for someone who is not associated with any
>company or other interests to do an objective test of the different
>solutions.
>Maybe this hasn't happened due to the complexity that Ed refers to here.
>
>Ken
>
>Ed McNierney wrote:
>
>
>
>>Rich -
>>
>>Are you aware of any benchmarks that compare ArcIMS to anything?
>>Benchmarking Internet map servers is a extremely complicated subject,
>>and I have not seen anyone make a serious attempt at doing it for ANY
>>products.
>>
>>        - Ed
>>
>>Ed McNierney
>>President and Chief Mapmaker
>>TopoZone.com / Maps a la carte, Inc.
>>73 Princeton Street, Suite 305
>>North Chelmsford, MA  01863
>>ed at topozone.com
>>(978) 251-4242
>>
>>-----Original Message-----
>>From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]
>>On Behalf Of Rich Binns
>>Sent: Monday, August 16, 2004 1:52 PM
>>To: MAPSERVER-USERS at LISTS.UMN.EDU
>>Subject: [UMN_MAPSERVER-USERS] Benchmarks
>>
>>Was wondering if anyone could direct me to where I could find
>>benchmark info comparing Mapserver to proprietary solutions,
>>
>>
>specifically ArcIMS.
>
>
>
 

--
----------------------------
CCGIS Christl & Stamm GbR
----------------------------
http://www.ccgis.de
http://www.mapbender.org
----------------------------
Service: +49 (0)228 90826 0
Direct:  +49 (0)228 90826 23
Fax:     +49 (0)228 90826 11
----------------------------
CCGIS Christl & Stamm GbR
Siemensstraße 8
53121 Bonn
GERMANY

<mailto:jacob.delfos at maunsell.com>  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20040818/4ea30964/attachment.htm>