<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<META content="MSHTML 6.00.2800.1458" name=GENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY bgColor=#ffffff>
<DIV dir=ltr align=left><SPAN class=359210414-18082004><FONT face=Arial
color=#0000ff size=2>Jacob -</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=359210414-18082004><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=359210414-18082004><FONT face=Arial
color=#0000ff size=2>Things like the amount of RAM and the physical disk
hardware can make a big difference in performance, so a different machine may
not produce "slightly" different results.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=359210414-18082004><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=359210414-18082004><FONT face=Arial
color=#0000ff size=2>I'm not saying that benchmarks aren't reproducible.
I'm saying that they are so dependent on the complete system
hardware/software/data/application configuration that it's very hard to compare
DIFFERENT systems. Knowing that MapServer is faster or slower than Product
X by 10% on a very specific configuration tells me very little about how it will
perform on MY hardware and MY configuration, which is usually what users are
interested in when using benchmark tests.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=359210414-18082004><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=359210414-18082004> <FONT
face=Arial color=#0000ff size=2>- Ed</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=359210414-18082004><!-- Converted from text/plain format -->
<P><FONT size=2>Ed McNierney<BR>President and Chief Mapmaker<BR>TopoZone.com /
Maps a la carte, Inc.<BR>73 Princeton Street, Suite 305<BR>North Chelmsford,
MA 01863<BR>Phone: +1 978 251-4242 Fax: +1 978 251-1396
</FONT></P></SPAN></DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> UMN MapServer Users List
[mailto:MAPSERVER-USERS@LISTS.UMN.EDU] <B>On Behalf Of </B>Jacob
Delfos<BR><B>Sent:</B> Tuesday, August 17, 2004 7:30 PM<BR><B>To:</B>
MAPSERVER-USERS@LISTS.UMN.EDU<BR><B>Subject:</B> Re: [UMN_MAPSERVER-USERS]
Benchmarks<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><FONT face=Arial size=2>I think if you take 1 server with a generic setup,
install both systems, load the same datasets (e.g. a roadnetwork only, or an
image only), and load the webpage 20 times from a browser on that same server,
or a sole machine connected to the server (crossover), you'd get a pretty good
idea, isn't it? You'd be able to say the same dataset with the same extents took
x times longer on system y. A different machine may produce slightly different
results, depending on different bottlenecks, but I'd say it would be reasonably
similar.</FONT></DIV>
<DIV> </DIV>
<DIV><FONT face=Arial size=2>-----Original Message-----<BR>From: UMN MapServer
Users List [mailto:MAPSERVER-USERS@LISTS.UMN.EDU] On Behalf Of Arnulf
Christl<BR>Sent: 17 August 2004 15:55<BR>To: <A
href="mailto:MAPSERVER-USERS@LISTS.UMN.EDU">MAPSERVER-USERS@LISTS.UMN.EDU</A><BR>Subject:
Re: [UMN_MAPSERVER-USERS] Benchmarks</FONT></DIV>
<DIV> </DIV><FONT face=Arial size=2>
<DIV><BR>Hi,<BR>the questions of benchmarking comes up every now an then. There
has been<BR>quite a lot of additional communication going on, some of which
the<BR>posters decided to keep off the list because of all the implications
Ed<BR>implicated...<BR>:-)<BR>Maybe we should gather some postings covering
aspects why it is<BR>pracitcally impossible to produce meaningful and verifiable
benchmarks.<BR>I remember that somebody wanted to do that but lost track what
came out<BR>of it. Does anybody know or is interested?</DIV>
<DIV> </DIV>
<DIV>Basically it burns down to the problem that geodata and requirements
are<BR>too heterogeneous to be comparable so that you can always compare
only<BR>one dataset with one specific question in mind (the complexity
issue).</DIV>
<DIV> </DIV>
<DIV>Arnulf.</DIV>
<DIV> </DIV>
<DIV>Ed McNierney wrote:</DIV>
<DIV> </DIV>
<DIV>>Ken -<BR>><BR>>I think the complexity is one issue. I think
the other issue is that<BR>>these are expensive products, and it's hard to
see an independent lab<BR>>doing the tests unless the vendors donate their
software. And if ESRI<BR>>is unwilling to publish the results of the
tests they control, I'd doubt<BR>>very much that they'd be willing to give
free software so someone else<BR>>could publish tests they DON'T
control!<BR>><BR>> -
Ed<BR>><BR>>-----Original Message-----<BR>>From: UMN MapServer Users
List [mailto:MAPSERVER-USERS@LISTS.UMN.EDU] On<BR>>Behalf Of Ken
Waters<BR>>Sent: Monday, August 16, 2004 2:33 PM<BR>>To: <A
href="mailto:MAPSERVER-USERS@LISTS.UMN.EDU">MAPSERVER-USERS@LISTS.UMN.EDU</A><BR>>Subject:
Re: [UMN_MAPSERVER-USERS] Benchmarks<BR>><BR>>All,<BR>><BR>>Last
weekend I was at an ESRI seminar, in conjunction with their
annual<BR>>conference. Somehow this question came up. The ESRI
"unofficial"<BR>>answer was that they have such statistics but have elected
to not share<BR>>them. Fair enough.<BR>><BR>>I think it would be
great for someone who is not associated with any<BR>>company or other
interests to do an objective test of the
different<BR>>solutions.<BR>>Maybe this hasn't happened due to the
complexity that Ed refers to here.<BR>><BR>>Ken<BR>><BR>>Ed
McNierney wrote:<BR>><BR>><BR>><BR>>>Rich
-<BR>>><BR>>>Are you aware of any benchmarks that compare ArcIMS to
anything?<BR>>>Benchmarking Internet map servers is a extremely
complicated subject,<BR>>>and I have not seen anyone make a serious
attempt at doing it for
ANY<BR>>>products.<BR>>><BR>>>
- Ed<BR>>><BR>>>Ed McNierney<BR>>>President and Chief
Mapmaker<BR>>>TopoZone.com / Maps a la carte, Inc.<BR>>>73 Princeton
Street, Suite 305<BR>>>North Chelmsford, MA
01863<BR>>>ed@topozone.com<BR>>>(978)
251-4242<BR>>><BR>>>-----Original Message-----<BR>>>From: UMN
MapServer Users List [mailto:MAPSERVER-USERS@LISTS.UMN.EDU]<BR>>>On Behalf
Of Rich Binns<BR>>>Sent: Monday, August 16, 2004 1:52 PM<BR>>>To: <A
href="mailto:MAPSERVER-USERS@LISTS.UMN.EDU">MAPSERVER-USERS@LISTS.UMN.EDU</A><BR>>>Subject:
[UMN_MAPSERVER-USERS] Benchmarks<BR>>><BR>>>Was wondering if anyone
could direct me to where I could find<BR>>>benchmark info comparing
Mapserver to proprietary solutions,<BR>>><BR>>><BR>>specifically
ArcIMS.<BR>><BR>><BR>></DIV>
<DIV> </DIV>
<DIV><BR>--<BR>----------------------------<BR>CCGIS Christl & Stamm
GbR<BR>----------------------------<BR><A
href="http://www.ccgis.de">http://www.ccgis.de</A><BR><A
href="http://www.mapbender.org">http://www.mapbender.org</A><BR>----------------------------<BR>Service:
+49 (0)228 90826 0<BR>Direct: +49 (0)228 90826
23<BR>Fax: +49 (0)228 90826
11<BR>----------------------------<BR>CCGIS Christl & Stamm
GbR<BR>Siemensstraße 8<BR>53121 Bonn<BR>GERMANY<BR></FONT></DIV>
<DIV><FONT face=Arial size=2><A
href="mailto:jacob.delfos@maunsell.com"></A></FONT> </DIV></BODY></HTML>