Benchmarks
Sean Gillies
sgillies at FRII.COM
Wed Aug 18 10:13:54 EDT 2004
Jacob, Ed:
I think the other problem would be the issue of performance
optimizations.
We MapServer users use quadtree indexes for shapefile performance, while
ArcIMS users, IIRC, need SDE for performance-enhancing spatial
indexing. There
are similiar issues with rasters re TIFF file pyramids/overviews, right?
They need completely different test fixtures with specific hardware
requirements.
It becomes a comparison of apples and oranges almost immediately ...
cheers,
Sean
On Aug 18, 2004, at 8:07 AM, Ed McNierney wrote:
> This is a multi-part message in MIME format.
>
> ------_=_NextPart_001_01C4852C.AC5FD6E6
> Content-Type: text/plain;
> charset="iso-8859-1"
> Content-Transfer-Encoding: quoted-printable
>
> Jacob -
> =20
> Things like the amount of RAM and the physical disk hardware can make
> a =
> big difference in performance, so a different machine may not produce =
> "slightly" different results.
> =20
> I'm not saying that benchmarks aren't reproducible. I'm saying that =
> they are so dependent on the complete system =
> hardware/software/data/application configuration that it's very hard
> to =
> compare DIFFERENT systems. Knowing that MapServer is faster or slower
> =
> than Product X by 10% on a very specific configuration tells me very =
> little about how it will perform on MY hardware and MY configuration, =
> which is usually what users are interested in when using benchmark =
> tests.
> =20
> - Ed
> Ed McNierney
> President and Chief Mapmaker
> TopoZone.com / Maps a la carte, Inc.
> 73 Princeton Street, Suite 305
> North Chelmsford, MA 01863
> Phone: +1 978 251-4242 Fax: +1 978 251-1396=20
>
>
> ________________________________
>
> From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]
> On =
> Behalf Of Jacob Delfos
> Sent: Tuesday, August 17, 2004 7:30 PM
> To: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: Re: [UMN_MAPSERVER-USERS] Benchmarks
>
>
> I think if you take 1 server with a generic setup, install both
> systems, =
> load the same datasets (e.g. a roadnetwork only, or an image only),
> and =
> load the webpage 20 times from a browser on that same server, or a
> sole =
> machine connected to the server (crossover), you'd get a pretty good =
> idea, isn't it? You'd be able to say the same dataset with the same =
> extents took x times longer on system y. A different machine may
> produce =
> slightly different results, depending on different bottlenecks, but
> I'd =
> say it would be reasonably similar.
> =20
> -----Original Message-----
> From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]
> On =
> Behalf Of Arnulf Christl
> Sent: 17 August 2004 15:55
> To: MAPSERVER-USERS at LISTS.UMN.EDU
> Subject: Re: [UMN_MAPSERVER-USERS] Benchmarks
> =20
>
> Hi,
> the questions of benchmarking comes up every now an then. There has
> been
> quite a lot of additional communication going on, some of which the
> posters decided to keep off the list because of all the implications Ed
> implicated...
> :-)
> Maybe we should gather some postings covering aspects why it is
> pracitcally impossible to produce meaningful and verifiable benchmarks.
> I remember that somebody wanted to do that but lost track what came out
> of it. Does anybody know or is interested?
> =20
> Basically it burns down to the problem that geodata and requirements
> are
> too heterogeneous to be comparable so that you can always compare only
> one dataset with one specific question in mind (the complexity issue).
> =20
> Arnulf.
> =20
> Ed McNierney wrote:
> =20
>> Ken -
>>
>> I think the complexity is one issue. I think the other issue is that
>> these are expensive products, and it's hard to see an independent lab
>> doing the tests unless the vendors donate their software. And if ESRI
>> is unwilling to publish the results of the tests they control, I'd =
> doubt
>> very much that they'd be willing to give free software so someone else
>> could publish tests they DON'T control!
>>
>> - Ed
>>
>> -----Original Message-----
>> From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]
>> =
> On
>> Behalf Of Ken Waters
>> Sent: Monday, August 16, 2004 2:33 PM
>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>> Subject: Re: [UMN_MAPSERVER-USERS] Benchmarks
>>
>> All,
>>
>> Last weekend I was at an ESRI seminar, in conjunction with their
>> annual
>> conference. Somehow this question came up. The ESRI "unofficial"
>> answer was that they have such statistics but have elected to not
>> share
>> them. Fair enough.
>>
>> I think it would be great for someone who is not associated with any
>> company or other interests to do an objective test of the different
>> solutions.
>> Maybe this hasn't happened due to the complexity that Ed refers to =
> here.
>>
>> Ken
>>
>> Ed McNierney wrote:
>>
>>
>>
>>> Rich -
>>>
>>> Are you aware of any benchmarks that compare ArcIMS to anything?
>>> Benchmarking Internet map servers is a extremely complicated subject,
>>> and I have not seen anyone make a serious attempt at doing it for ANY
>>> products.
>>>
>>> - Ed
>>>
>>> Ed McNierney
>>> President and Chief Mapmaker
>>> TopoZone.com / Maps a la carte, Inc.
>>> 73 Princeton Street, Suite 305
>>> North Chelmsford, MA 01863
>>> ed at topozone.com
>>> (978) 251-4242
>>>
>>> -----Original Message-----
>>> From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]
>>> On Behalf Of Rich Binns
>>> Sent: Monday, August 16, 2004 1:52 PM
>>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>>> Subject: [UMN_MAPSERVER-USERS] Benchmarks
>>>
>>> Was wondering if anyone could direct me to where I could find
>>> benchmark info comparing Mapserver to proprietary solutions,
>>>
>>>
>> specifically ArcIMS.
>>
>>
>>
> =20
>
> --
> ----------------------------
> CCGIS Christl & Stamm GbR
> ----------------------------
> http://www.ccgis.de
> http://www.mapbender.org
> ----------------------------
> Service: +49 (0)228 90826 0
> Direct: +49 (0)228 90826 23
> Fax: +49 (0)228 90826 11
> ----------------------------
> CCGIS Christl & Stamm GbR
> Siemensstra=DFe 8
> 53121 Bonn
> GERMANY
>
> <mailto:jacob.delfos at maunsell.com> =20
>
> ------_=_NextPart_001_01C4852C.AC5FD6E6
> Content-Type: text/html;
> charset="iso-8859-1"
> Content-Transfer-Encoding: quoted-printable
>
> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
> <HTML><HEAD>
> <META http-equiv=3DContent-Type content=3D"text/html; =
> charset=3Diso-8859-1">
> <META content=3D"MSHTML 6.00.2800.1458" name=3DGENERATOR>
> <STYLE></STYLE>
> </HEAD>
> <BODY bgColor=3D#ffffff>
> <DIV dir=3Dltr align=3Dleft><SPAN class=3D359210414-18082004><FONT =
> face=3DArial=20
> color=3D#0000ff size=3D2>Jacob -</FONT></SPAN></DIV>
> <DIV dir=3Dltr align=3Dleft><SPAN class=3D359210414-18082004><FONT =
> face=3DArial=20
> color=3D#0000ff size=3D2></FONT></SPAN> </DIV>
> <DIV dir=3Dltr align=3Dleft><SPAN class=3D359210414-18082004><FONT =
> face=3DArial=20
> color=3D#0000ff size=3D2>Things like the amount of RAM and the
> physical =
> disk=20
> hardware can make a big difference in performance, so a different =
> machine may=20
> not produce "slightly" different results.</FONT></SPAN></DIV>
> <DIV dir=3Dltr align=3Dleft><SPAN class=3D359210414-18082004><FONT =
> face=3DArial=20
> color=3D#0000ff size=3D2></FONT></SPAN> </DIV>
> <DIV dir=3Dltr align=3Dleft><SPAN class=3D359210414-18082004><FONT =
> face=3DArial=20
> color=3D#0000ff size=3D2>I'm not saying that benchmarks aren't =
> reproducible. =20
> I'm saying that they are so dependent on the complete system=20
> hardware/software/data/application configuration that it's very hard
> to =
> compare=20
> DIFFERENT systems. Knowing that MapServer is faster or slower
> than =
> Product=20
> X by 10% on a very specific configuration tells me very little about
> how =
> it will=20
> perform on MY hardware and MY configuration, which is usually what
> users =
> are=20
> interested in when using benchmark tests.</FONT></SPAN></DIV>
> <DIV dir=3Dltr align=3Dleft><SPAN class=3D359210414-18082004><FONT =
> face=3DArial=20
> color=3D#0000ff size=3D2></FONT></SPAN> </DIV>
> <DIV dir=3Dltr align=3Dleft><SPAN =
> class=3D359210414-18082004> <FONT=20
> face=3DArial color=3D#0000ff size=3D2>- Ed</FONT></SPAN></DIV>
> <DIV dir=3Dltr align=3Dleft><SPAN class=3D359210414-18082004><!-- =
> Converted from text/plain format -->
> <P><FONT size=3D2>Ed McNierney<BR>President and Chief =
> Mapmaker<BR>TopoZone.com /=20
> Maps a la carte, Inc.<BR>73 Princeton Street, Suite 305<BR>North =
> Chelmsford,=20
> MA 01863<BR>Phone: +1 978 251-4242 Fax: +1 978 =
> 251-1396=20
> </FONT></P></SPAN></DIV><BR>
> <DIV class=3DOutlookMessageHeader lang=3Den-us dir=3Dltr align=3Dleft>
> <HR tabIndex=3D-1>
> <FONT face=3DTahoma size=3D2><B>From:</B> UMN MapServer Users List=20
> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] <B>On Behalf Of </B>Jacob=20
> Delfos<BR><B>Sent:</B> Tuesday, August 17, 2004 7:30
> PM<BR><B>To:</B>=20
> MAPSERVER-USERS at LISTS.UMN.EDU<BR><B>Subject:</B> Re: =
> [UMN_MAPSERVER-USERS]=20
> Benchmarks<BR></FONT><BR></DIV>
> <DIV></DIV>
> <DIV><FONT face=3DArial size=3D2>I think if you take 1 server with a =
> generic setup,=20
> install both systems, load the same datasets (e.g. a roadnetwork only,
> =
> or an=20
> image only), and load the webpage 20 times from a browser on that same
> =
> server,=20
> or a sole machine connected to the server (crossover), you'd get a =
> pretty good=20
> idea, isn't it? You'd be able to say the same dataset with the same =
> extents took=20
> x times longer on system y. A different machine may produce slightly =
> different=20
> results, depending on different bottlenecks, but I'd say it would be =
> reasonably=20
> similar.</FONT></DIV>
> <DIV> </DIV>
> <DIV><FONT face=3DArial size=3D2>-----Original Message-----<BR>From:
> UMN =
> MapServer=20
> Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On Behalf Of
> Arnulf=20
> Christl<BR>Sent: 17 August 2004 15:55<BR>To: <A=20
> href=3D"mailto:MAPSERVER-USERS at LISTS.UMN.EDU">MAPSERVER-
> USERS at LISTS.UMN.E=
> DU</A><BR>Subject:=20
> Re: [UMN_MAPSERVER-USERS] Benchmarks</FONT></DIV>
> <DIV> </DIV><FONT face=3DArial size=3D2>
> <DIV><BR>Hi,<BR>the questions of benchmarking comes up every now an =
> then. There=20
> has been<BR>quite a lot of additional communication going on, some of =
> which=20
> the<BR>posters decided to keep off the list because of all the =
> implications=20
> Ed<BR>implicated...<BR>:-)<BR>Maybe we should gather some postings =
> covering=20
> aspects why it is<BR>pracitcally impossible to produce meaningful and =
> verifiable=20
> benchmarks.<BR>I remember that somebody wanted to do that but lost
> track =
> what=20
> came out<BR>of it. Does anybody know or is interested?</DIV>
> <DIV> </DIV>
> <DIV>Basically it burns down to the problem that geodata and =
> requirements=20
> are<BR>too heterogeneous to be comparable so that you can always
> compare =
>
> only<BR>one dataset with one specific question in mind (the
> complexity=20
> issue).</DIV>
> <DIV> </DIV>
> <DIV>Arnulf.</DIV>
> <DIV> </DIV>
> <DIV>Ed McNierney wrote:</DIV>
> <DIV> </DIV>
> <DIV>>Ken -<BR>><BR>>I think the complexity is one
> issue. =
> I think=20
> the other issue is that<BR>>these are expensive products, and it's =
> hard to=20
> see an independent lab<BR>>doing the tests unless the vendors
> donate =
> their=20
> software. And if ESRI<BR>>is unwilling to publish the results
> =
> of the=20
> tests they control, I'd doubt<BR>>very much that they'd be willing
> to =
> give=20
> free software so someone else<BR>>could publish tests they DON'T=20
> control!<BR>><BR>> -=20
> Ed<BR>><BR>>-----Original Message-----<BR>>From: UMN
> MapServer =
> Users=20
> List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On<BR>>Behalf Of Ken=20
> Waters<BR>>Sent: Monday, August 16, 2004 2:33 PM<BR>>To: <A=20
> href=3D"mailto:MAPSERVER-USERS at LISTS.UMN.EDU">MAPSERVER-
> USERS at LISTS.UMN.E=
> DU</A><BR>>Subject:=20
> Re: [UMN_MAPSERVER-USERS] =
> Benchmarks<BR>><BR>>All,<BR>><BR>>Last=20
> weekend I was at an ESRI seminar, in conjunction with their=20
> annual<BR>>conference. Somehow this question came up.
> The =
> ESRI=20
> "unofficial"<BR>>answer was that they have such statistics but have
> =
> elected=20
> to not share<BR>>them. Fair enough.<BR>><BR>>I think it
> =
> would be=20
> great for someone who is not associated with any<BR>>company or
> other =
>
> interests to do an objective test of the=20
> different<BR>>solutions.<BR>>Maybe this hasn't happened due to
> the =
>
> complexity that Ed refers to
> here.<BR>><BR>>Ken<BR>><BR>>Ed=20
> McNierney wrote:<BR>><BR>><BR>><BR>>>Rich=20
> -<BR>>><BR>>>Are you aware of any benchmarks that compare =
> ArcIMS to=20
> anything?<BR>>>Benchmarking Internet map servers is a
> extremely=20
> complicated subject,<BR>>>and I have not seen anyone make a =
> serious=20
> attempt at doing it for=20
> ANY<BR>>>products.<BR>>><BR>>>
> &=
> nbsp; =20
> - Ed<BR>>><BR>>>Ed McNierney<BR>>>President and
> Chief=20
> Mapmaker<BR>>>TopoZone.com / Maps a la carte, Inc.<BR>>>73
> =
> Princeton=20
> Street, Suite 305<BR>>>North Chelmsford, MA =20
> 01863<BR>>>ed at topozone.com<BR>>>(978)=20
> 251-4242<BR>>><BR>>>-----Original =
> Message-----<BR>>>From: UMN=20
> MapServer Users List =
> [mailto:MAPSERVER-USERS at LISTS.UMN.EDU]<BR>>>On Behalf=20
> Of Rich Binns<BR>>>Sent: Monday, August 16, 2004 1:52 =
> PM<BR>>>To: <A=20
> href=3D"mailto:MAPSERVER-USERS at LISTS.UMN.EDU">MAPSERVER-
> USERS at LISTS.UMN.E=
> DU</A><BR>>>Subject:=20
> [UMN_MAPSERVER-USERS] Benchmarks<BR>>><BR>>>Was wondering
> if =
> anyone=20
> could direct me to where I could find<BR>>>benchmark info =
> comparing=20
> Mapserver to proprietary =
> solutions,<BR>>><BR>>><BR>>specifically=20
> ArcIMS.<BR>><BR>><BR>></DIV>
> <DIV> </DIV>
> <DIV><BR>--<BR>----------------------------<BR>CCGIS Christl &
> Stamm =
>
> GbR<BR>----------------------------<BR><A=20
> href=3D"http://www.ccgis.de">http://www.ccgis.de</A><BR><A=20
> href=3D"http://www.mapbender.org">http://www.mapbender.org</
> A><BR>-------=
> ---------------------<BR>Service:=20
> +49 (0)228 90826 0<BR>Direct: +49 (0)228 90826=20
> 23<BR>Fax: +49 (0)228 90826=20
> 11<BR>----------------------------<BR>CCGIS Christl & Stamm=20
> GbR<BR>Siemensstra=DFe 8<BR>53121 Bonn<BR>GERMANY<BR></FONT></DIV>
> <DIV><FONT face=3DArial size=3D2><A=20
> href=3D"mailto:jacob.delfos at maunsell.com"></A></FONT> </DIV></
> BODY><=
> /HTML>
>
> ------_=_NextPart_001_01C4852C.AC5FD6E6--
>
More information about the mapserver-users
mailing list