[Mapserver-users] Slow MapServer with large data

Wed May 14 03:18:51 PDT 2003

Hi all,

This is my first appearance on this list and I must say that I am very impressed! Thank you guys for your contributions!!
This time I give some more details of our system and I try to summarize the input so far together with some further questions.

I can see that a short description of our system maybe will make my needs more clearly (sorry I didn’t bring this the first time...): 
- Our system is integrated with a search-engine, that looks for relevant GIS-data among 17.000 layers. 
- The data are in shape and tab-format (and coverage but we’re not handling that right now) and has every kind of geo-types and sizes. The data are regularly updated. 
- By clicking on a search-result (or by choosing a shape/tab-file from a pathfinder in the browser) our system is reading extent and geotype for the chosen layer, generating a mapfile dynamically and MapServer (PHP Mapscript) generates a map.
- We are always showing one layer at a time and always the complete layer from the start.
- For now we’re not having any queries, but we would like to make it possible to see metadata for the data. 

There have been some good advices. Here is a short summary and some more questions:

1. Use shapetree.
JC:  
I could test every request to see if a qix-file already exists and make it if it doesn’t. 
However I have tried shptree on a shapefile (25 MB, DBF-file 36MB) and a qix-file by 11,5 MB was created. But there was no change in speed: It took 15-20 sec. (and so did the shptree-command).
Is any change needed in mapfile or Mapscript-code – or is MapServer finding the qix-file itself? 
Is it possible to test if the qix-file really is used?

2. Use minscale/maxscale to show less detailed maps when showing the complete layer. The maps could be created in (at least) three ways:
      - thin out data with ArcView tools or another tool?
      - use a raster “screenshot” created with shp2img.
      - use “postgis and download the simplify function that implements the DP-algorithm 
(http://cobalt.rmnet.it/~san/postgis/simplify/simplify-0.2.tgz.”
JC: 
It is only the large files that should be thinned out – I am not interested in small files getting lesser detailed. It is maybe possible to check the filesize and only do the “thinning” and minscale/maxscale on large files. Howewer the thinning (or screenshot-making) should be done in runtime because the data is updated regularly. 
Is this possible?
Does anyone know the tool to thin out shape-files, that Arnulf mentioned in his second mail? (I haven’t been looking for it myself yet, but just in case someone have it right by ....)

3.  Throw the shapes into PG/PostGIS.  
JC:  
Is this the same solution as nr 2.3?

4. “Separate the MapServer software and the PostGreSQL DB to run on separate machines
 (hardware)”. 
JC: 
Could be a idea later on

5. Convert MapInfo files to shapefiles (and then use shapetree etc.)
JC: 
This should also be done in runtime (first time the file is read) – and then followed by shapetree etc. from above. 
How to do it anyway? 

All in all I must admit: It seems a lot more triggy to deal with my speed-problems than I expected. 
If anyone have other ideas I’m still all ears... (could you say that in english?)

cheers

Jan Christiansen

> On 9 May 2003 at 13:14, jankarin at post1.dknet.dk wrote:
> 
> > Hello
> > 
> > I have a little problem with the speed in MapServer. 
> > We're trying to make qiuck-viewer to shape- and
> tab-formats, but 
> > MapServer is running slow with large data: 20 seconds:
> Shapefil 17 MB,
> > dbf-fil 9,5 MB. 45 seconds: Map-fil 31 MB, Dat-fil 25 MB.
> > 
> > I have been trying with the GCI-version and PHP Mapscript
> - same
> > result.
> > 
> > I am running MapServer 3.6.4 with PNG-output under Apache
> 2.4.5 on
> > Windows 2000. The pc is Pentim III, 650 Mhz, 262 MB RAM.
> > 
> > In another installation (PII, 400 Mhz, 196 MB RAM) a
> shapefile
> > (shapefile 17 MB, dbf-fil 9,5 MB) took 80 seconds to
> show.
> > 
> > We're using MapServer in an application which is
> dynamically creating
> > map-files with Mapscript (only showing one layer at a
> time). We're
> > talknig about 17.000 layers in shape- and tabformats, so
> it is not
> > possible to improve indexfiles or anything else related to
> a single
> > layer.
> > 
> > Can anyone help?
> > 
> > Thanks,
> > 
> > Jan Christiansen
> > jankarin at post1.dknet.dk
> > Engineering College of Copenhagen
> > 
> > _______________________________________________
> > Mapserver-users mailing list
> > Mapserver-users at lists.gis.umn.edu
> > http://lists.gis.umn.edu/mailman/listinfo/mapserver-users
> > 
> 
> 