New mapping application

Tue Nov 28 16:55:57 EST 2000

Under my test load, the system never swapped out actually. Everything was
running in RAM. The CPU's were Pentium II Deschutes.

Well, it's the morning after and things went OK overall. The server actually
got brutally attacked ! (Almost crashed ! Linux ! Can it be ?)
The OS started running out of File Handles, so I had to reduce the number of
Apache clients, that helped.

So far as I can tell, over the past 26 hours, 80 000 (Yes, eighty THOUSAND)
maps were delivered to the clients ! We resleased results at 10PM local
time, and the server was so slow it was barely usable for about 1.5 hours.
After that things started running better. The first value of the load
average was at 90 for a while ! It may have gone higher, I wasn't paying
much attention to it. Do note that despite all this, it still only swapped
out a couple of megs I think. My contention is that CPU power WAS the
bottleneck here.

But Mapserver ran fine, service, although very slow at times, was allways
available (Unlike someother big canadians sites that completely crashed,
like elections.ca and cbc.ca).

Do note I was also using a heavily customized Apache !

No special data preprocessing was done. We were using standard shapefiles
form our usual dataset (National Atlas 7.5 Million, available from Geogratis
at http://geogratis.ccrs.nrcan.gc.ca), although I think some of the layers I
had were modified and adapted to work at different zoom levels, some are
more "dense" than others, have more features, but none were generalized
specifically for this task. This is the same data Dan uses on the gm75 demo
BTW.

Cheers,
J.F.

> ----------
> From: 	Stephen Lime[SMTP:steve.lime at dnr.state.mn.us]
> Sent: 	Tuesday, November 28, 2000 4:17 PM
> To: 	Jean-Francois.Doyon at CCRS.NRCan.gc.ca; jrf at segovia.mit.edu
> Cc: 	mapserver-users at lists.gis.umn.edu
> Subject: 	RE: New mapping application
> 
> My experience is that the MapServer tends to be IO bound, not memory
> bound. You can only force so many disk accesses at once... My guess is
> that with this type of application throwing more processors at the problem
> does little to boost performance.
> 
> I'm curious what, if any, data preprocessing was done to expedite
> rendering? (i.e. data thining, pre-projection, spatial indexes)
> 
> Steve
> 
> Stephen Lime
> Internet Applications Analyst
> 
> Minnesota DNR
> 500 Lafayette Road
> St. Paul, MN 55155
> 651-297-2937
> 
> >>> John Frank <jrf at segovia.mit.edu> 11/28/00 01:40PM >>>
> JF-
> 
> Does mapserver use all the RAM?  "100 concurrent clients" means its
> working on 100 maps simultaneously, right?  With 100 maps rendering at any
> moment, I think that would be about 420MB.
> 
> Is your "Quad 400" a four-processor machine with 400MHz Athlons?  Or
> something else with a smaller cache?
> 
> What is the bottleneck that increases the map delivery speed to 5-7
> seconds?  Is it processor limited?
> 
> John
> 
> 
> 
> 
> 
> 
> 
> On Mon, 27 Nov 2000, Doyon, Jean-Francois wrote:
> 
> > actually the tool will remain available in the future, we've got it ,
> might
> > as well use it :)
> > 
> > Yes, I hope so too, I'm not too worried though, things don't work quite
> the
> > same here, and
> > we're not stuck with only two choices :)
> > 
> > As for the load testing:
> > 
> > On a Quad 400 with 512MB RAM and a RAID 0 Storage Sub-System, I got up
> to
> > 250 maps/minute with 100 concurrent clients pouding the box, generating
> maps
> > continuously. I was VERY happy with that result :) This was using the
> > Election Interface/Map you've seen. There was no randomizing (Allways
> the
> > default, full extent map, which basically is the worst case scenario).
> Under
> > this load the map took about 5-7 seconds to show up.
> > 
> > The mapscript versions worked well also. I don't have a number of maps
> on
> > that, but I do know every once in a while an Apache child would bomb
> with a
> > sig11 (tail -f'ing the error log). I suppose this is a SWIG thing ...
> Both
> > Perl and PHP did this. I can't really compare to the CGI since they were
> > different boxes.
> > 
> > The server is RH 7.0 with the CVS mapserver.
> > 
> > I was using Apache's JMeter to create the load, very cool tool.
> > (http://java.apache.org/jmeter)
> > 
> > Overall the CGI version gets my thumbs up, wayyyyyyy up. Another tool we
> > use, died at 20 maps/minute on a dual 400 under NT4. Open Source wins
> yet
> > again :)
> > 
> > Sorry if this all seems add-hoc. The intent was never to do a truly
> > disciplined benchmark, just to get an idea of perfromance and stability
> > under load. Maybe I'll do somehting with a more "scientific" approach
> > someday. Although if anybody out there cares, just get the tool I
> > mentionned, makes the testing real easy :)
> > 
> > Cheers,
> > J.F.
> > 
> > > ----------
> > > From: 	Stephen Lime[SMTP:steve.lime at dnr.state.mn.us] 
> > > Sent: 	Monday, November 27, 2000 4:23 PM
> > > To: 	Jean-Francois.Doyon at CCRS.NRCan.gc.ca;
> > > 'mapserver-users at lists.gis.umn.edu' 
> > > Subject: 	Re: New mapping application
> > > 
> > > Sweet. Hopefully your elections will go smoother than in the states.
> ;-) I
> > > really like
> > > the application in general- election results. I assume this will have
> a
> > > limited shelf so
> > > it's not worth adding to the demo page.
> > > 
> > > I know you were doing some stress testing. How'd that come out?
> > > 
> > > Steve
> > > 
> > > Yes, the URL parameter switching is real handy. Use it all the time
> > > myself.
> > > 
> > > Stephen Lime
> > > Internet Applications Analyst
> > > 
> > > Minnesota DNR
> > > 500 Lafayette Road
> > > St. Paul, MN 55155
> > > 651-297-2937
> > > 
> > > >>> "Doyon, Jean-Francois" <Jean-Francois.Doyon at CCRS.NRCan.gc.ca>
> 11/27/00
> > > 11:24AM >>>
> > > Goodday,
> > > 
> > > I just wanted to give everyone a heads-up.
> > > 
> > > I've implemented a new mapping interface using Mapserver CGI to
> provide
> > > live
> > > up to the minute mapping of our federal election results as they come
> in.
> > > (Canadian Federal Election).
> > > 
> > > You will be able to find a link here:
> > > 
> > > http://atlas.gc.ca/english/facts/elections/elections2000/index.htm 
> > > 
> > > Click on Election 2000 Mapping.
> > > 
> > > There currently are only Screen-Shots, the interface will go live
> sometime
> > > this afternoon, with data showing up sometime shortly after 10pm EST.
> > > 
> > > Feedback is most welcome of course !
> > > 
> > > Sorry I've been so quiet lately, been working on this non-stop for a
> > > while.
> > > 
> > > Hopefully I'll get back into the swing of things soon now (Such as the
> > > documentation project).
> > > 
> > > And Stephen, that thing where you can set almost any Mapfile paramter
> with
> > > a
> > > structured URL parameter (map_layer_???) is very handy :)
> > > Great for implementing bi-lingual interfaces.
> > > 
> > > Cheers,
> > > 
> > > 
> > > Jean-Francois Doyon
> > > Internet Service Development and Systems Support
> > > GeoAccess Division
> > > Natural Resources Canada
> > > http://atlas.gc.ca 
> > > (613) 992-4902
> > > 
> > > 
> > 
> 
> 