[mapserver-users] Mapserver performance on apache - heavy load systems

Mon Dec 12 02:16:03 PST 2011

Hi,

Absolutely do not increase RAM, CPU or anything on that line. Instead, install all your data on a small slow laptop and make the system fast on that.

- Analyze your situaltion and think what you want to give for your customers
- Optimise the data for that purpose
- Final step is to acquire as much computing power as needed

Let's assume that you are going to publish your data through WMS.  You said you have 4 000 000 features and it takes a long time to load them. I believe it is slow but fortunately you do not need to do it. The computer screen with 1152x864 resolution has only 955 328 pixels so your full screen view is trying to illustrate on average four features per pixel. If someone orders you to draw a map by hand with a pen and paper in that way you would say it is lunatic and that's correct. The key to draw faster is to draw less.

This link shows a course example of speeding vector renderign by making Mapserver to draw less.
http://www.paikkatietoikkuna.fi/web/en/map-window?zoomLevel=6&coord=291620_6798042&mapLayers=base_35+100+!default!,34+100+&showMarker=false&forceCache=true

There is a scale dependent group layer "Lan Parcel Register 2010".  The group consists or the original data layer and two reduced sets. Original data is shown at scale 1:100000 or bigger (big scale - small number in scale denominator) and it comes from a shapefile with 1.2 million polygons.  In the application the switch to original data happens between named zoom levels "Town" and "Part of town". When zoomed out the data comes from teh first reduced data set (10% sample) and finally from the second reduced data set (1% sample).

In this case I made the ten percent sample and one percent sample by selecting parcel IDs ending to one zero or two zeros, respectively. This way I can broadcast the selected parcels uniformly on the whole map area for giving a hint about the places having parcels.  There is also a maximum limit for the number of parcels which are rendered (MAXFEATURES 10000). I considered that ten thousand parcels per screen is always enough.

For other data some other basis could be better for reducing the number of features to render.  For example it is usually enough to render at maximum a few hundred biggest lakes an a map and that can be done by sorting data to descending order according to area and then applying MAXFEATURES.

Another good way for making Mapserver to draw less and thus draw faster is to generalize the geometries to have less vertises. If you have a hundred pixels on a screen for showing a polygon then it will look just the same for the user with 20 or 20 000 vertises.

The mapfile used in the linked example is attached. It is a little bit modified and does not necesarily work straight ahead but it should show the idea.

Here is my ordered list about making vector rendering fast. Other people can suggest re-ordering or adding new items.  I suppose there is something to think about when using classified layers and selecting classes with classitems and expressions.

1. Check the spatial indexes for shapefiles and in the database
2. Check other database indexes. In 98 percent of cases when user says that database is slow it just does not have a proper index for the query.
3. Draw less - less features, less vertices
==============================
4. Draw simple - multiple styles and advanced labels have some cost
5. Re-project data beforehand instead of on-the-fly

-Jukka Rahkonen-

________________________________
Peter Maes  wrote:
Hi Steve,

Thanks for your reply. However, most of my datasources are vector data. I've created a spatial index on the shapefiles, but with 4 000 000 features it still takes a long time to load everything.

Should I increase ram, or cpu? or Both? the machines have 8 GB ram atm. I'm not sure what mapserver needs in heavy configurations.

Besides the page on mapserver.org<http://mapserver.org> i dont find much on improving performance for load intensive systems.

2011/12/9 Lime, Steve D (DNR) <Steve.Lime at state.mn.us<mailto:Steve.Lime at state.mn.us>>
Apache should be plenty quick. Start with using fastcgi and optimizing your data: e.g. create spatial indexes, use optimized raster formats and then optimizing your configuration (scale dependent layers w/things like reduced resolution layers where possible). -Steve

From: mapserver-users-bounces at lists.osgeo.org<mailto:mapserver-users-bounces at lists.osgeo.org> [mailto:mapserver-users-bounces at lists.osgeo.org<mailto:mapserver-users-bounces at lists.osgeo.org>] On Behalf Of Peter Maes
Sent: Friday, December 09, 2011 6:31 AM
To: mapserver-users at lists.osgeo.org<mailto:mapserver-users at lists.osgeo.org>
Subject: [mapserver-users] Mapserver performance on apache - heavy load systems

Hi all,

We are planning to launch a major mapserver production site where i'm required to launch many mapserver instances in some form of cluster.

So I basically have some questions related to this:

- Does anyone have good tips on performance for running mapserver on apache?
- Any specific build parameters I should pay attention to?
- Does it run as a native apache module, or is it required to work through cgi?  In which case: will i get better performance by using fastcgi? (we are not using databases but shapefiles).
- Should I consider lighttpd or nginx instead?

- Any other tips related to clustering mapserver instances? :).

Note that we are using live data, so we cannot cache them, except for a minute or so.

Thanks for your time.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20111212/59d4d9a9/attachment.htm>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Group_with_reduced_layers.txt
URL: <http://lists.osgeo.org/pipermail/mapserver-users/attachments/20111212/59d4d9a9/attachment.txt>