[Mapserver-users] Whole lotta shakin' going on

[Ed McNierney]

Arnulf -

Performance tuning is pretty complicated, and I'm always very reluctant
to tell someone "Here's the hardware we use and performance is great"
because it's my belief that the actual organization of the DATA and the
APPLICATION are much more important.

And serving large amounts of data (we're now in the 20 TB range with
MapServer) is often very misleading.  At any one moment the vast
majority of that data is sitting there on disk doing nothing (or at
least it SHOULD be doing nothing <g>), so what's the difference between
100GB of idle disk and 40TB of idle disk?

For serving large amounts of raster data, I think the rules are simple
to state (but not always easy to follow):

1. Make sure you have resampled the imagery (if necessary) so
small-scale (zoomed-out) views use a data source with input resolution.

2. Do not use compressed imagery unless you fully understand how the
decompression works.

3. Do not use compression schemes that require a LOT of data to be read
from disk in order to decode a small portion of the image.

4. Use good disk hardware (RAID-5) and organization.

5. Use lots of RAM so the server can both cache data and avoid swapping.

6. Tile the imagery into tiles that are of relatively modest size and
use tile indexes.  Do not break the tiles too small - most map requests
should require only one input tile to be read.

By far the most important part is to understand what's going on when the
user requests a map, and cut out the parts that aren't needed!

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: (978) 251-4242  Fax: (978) 251-1396
ed at topozone.com


-----Original Message-----
From: Arnulf Christl (CCGIS) [mailto:arnulf.christl at ccgis.de] 
Sent: Friday, September 05, 2003 4:23 AM
To: Ed McNierney; mapserver-users at lists.gis.umn.edu
Subject: AW: [Mapserver-users] Whole lotta shakin' going on

Hi,
thats great news, congratulations. Could you share some information
concerning the hardware and architecture you are using. We have
potential
users here who are very interested in using UMN MapServer but still fear
that performance might be a problem.
Its interesting right down to the details - whether to rather use
multiprocessor machines / which OS / amount of data served, etc.

We ourselves are more into vector graphics and use PostGIS with
MapServer
where it is possible to distribute the load on several machines in a
distributed architecture. But we lack the know-how for serving real big
amounts of raster data (talking about terabyte).

Any information would be welcome.

Regards, Arnulf.


-----Ursprungliche Nachricht-----
Von: mapserver-users-admin at lists.gis.umn.edu
[mailto:mapserver-users-admin at lists.gis.umn.edu]Im Auftrag von Ed
McNierney
Gesendet: Freitag, 5. September 2003 05:05
An: mapserver-users at lists.gis.umn.edu
Betreff: [Mapserver-users] Whole lotta shakin' going on


Folks -

As I posted on this list a week or so ago, we've migrated TopoZone over
to a 100%-MapServer site.  Everything's been doing fine, but we're still
keeping an eye on things.

About an hour ago I noticed a sudden huge spike in traffic on TopoZone.
At TopoZone we get a lot of users, so together the average usage forms a
very characteristic curve over the course of the day.  Weekdays it
starts climbing in the morning, leveling a bit around midday, reaching a
peak right around 4 PM Eastern time, then falling and climbing to a
lower, secondary peak around 9 - 10 PM Eastern time before dropping to
its 5 AM Eastern time low point.

At about 9:40 PM our traffic suddenly shot up to its highest level since
we made the switch to the MapServer site.  This was very strange.  When
I investigated, I found that actually almost all the traffic was being
handled by one of our eight MapServer systems.  Our data is distributed
geographically across these systems, and it turned out that we were
getting a huge number of requests for maps of the same area.

Hmmmm....  a hacker?  That was my first thought, but it would need to be
someone with a LOT of clear bandwidth.  The log files showed the traffic
was coming from a lot of different clients - an attempt at a coordinated
Denial of Service attack?

A bit more log-sniffing took me here:

http://earthquake.usgs.gov/recenteqsUS/Quakes/nc40146208.htm

There's a "topo map" link in the "Additional Information" section near
the bottom.  Mystery solved.

Things are still quite busy, but they've cooled off a bit.  I'm sure
there will be a bunch of curious users in the morning, too.

The one MapServer system that was taking all the heat was serving an
average of 10 custom topo maps a second over a full 10 minute period.
MapServer was chugging along just fine - the only "problem" was that I
got one complaint from Apache that I might want to increase my
MaxClients setting <g>.

	- Ed

Ed McNierney
President and Chief Mapmaker
TopoZone.com / Maps a la carte, Inc.
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: (978) 251-4242  Fax: (978) 251-1396
ed at topozone.com


_______________________________________________
Mapserver-users mailing list
Mapserver-users at lists.gis.umn.edu
http://lists.gis.umn.edu/mailman/listinfo/mapserver-users