[Mapserver-users] Slow MapServer with large data

Wed May 14 09:14:25 EDT 2003

Hi Ed

Yes, as it is we're always starting with the entire extent in the first view. 
The reason is that the user don't know to much about the layer, so the whole idea was to give a quick view of the map. With the search-engine and our "quick-viewer" the user should get an overview of posssible data to fulfill his needs - and from there open the chosen data in ArcView or MapInfo to further use. 

It could however be a good idea to display only a smaller part of data if the shape-file is large.

Jan

Ed McNierney <ed at topozone.com> wrote:
> Jan -
> 
> You mention that you always display "the complete layer from
> the start" for each map display, and I wanted to make sure
> we knew what that meant.  Do you always display each map so
> the ENTIRE extent of the layer is visible?  For example, if
> each layer were a country in Europe, would each request show
> a map of all of France, or Italy, or Poland, etc.?
> 
> If that is the case, you should not expect shptree to help. 
> Shptree indexes your data so MapServer can more quickly
> determine exactly what subset of the data needs to be drawn
> because it is visible on the requested map.  If you're going
> to draw ALL the data, there's no point in trying to optimize
> what portions of it need to be drawn.
> 
> 	- Ed
> 
> Ed McNierney
> President and Chief Mapmaker
> TopoZone.com / Maps a la carte, Inc.
> 73 Princeton Street, Suite 305
> North Chelmsford, MA  01863
> ed at topozone.com
> (978) 251-4242 
> 
> -----Original Message-----
> From: jankarin at post1.dknet.dk
> [mailto:jankarin at post1.dknet.dk]
> Sent: Wednesday, May 14, 2003 6:19 AM
> To: mapserver-users at lists.gis.umn.edu;
> woodbri at swoodbridge.com;
> arnulf.christl at ccg.is.de
> Subject: Re: Re: [Mapserver-users] Slow MapServer with large
> data
> 
> 
> Hi all,
> 
> This is my first appearance on this list and I must say that
> I am very impressed! Thank you guys for your
> contributions!!
> This time I give some more details of our system and I try
> to summarize the input so far together with some further
> questions.
> 
> I can see that a short description of our system maybe will
> make my needs more clearly (sorry I didn't bring this the
> first time...): 
> - Our system is integrated with a search-engine, that looks
> for relevant GIS-data among 17.000 layers. 
> - The data are in shape and tab-format (and coverage but
> we're not handling that right now) and has every kind of
> geo-types and sizes. The data are regularly updated. 
> - By clicking on a search-result (or by choosing a
> shape/tab-file from a pathfinder in the browser) our system
> is reading extent and geotype for the chosen layer,
> generating a mapfile dynamically and MapServer (PHP
> Mapscript) generates a map.
> - We are always showing one layer at a time and always the
> complete layer from the start.
> - For now we're not having any queries, but we would like to
> make it possible to see metadata for the data. 
> 
> There have been some good advices. Here is a short summary
> and some more questions:
> 
> 1. Use shapetree.
> JC:  
> I could test every request to see if a qix-file already
> exists and make it if it doesn't. 
> However I have tried shptree on a shapefile (25 MB, DBF-file
> 36MB) and a qix-file by 11,5 MB was created. But there was
> no change in speed: It took 15-20 sec. (and so did the
> shptree-command).
> Is any change needed in mapfile or Mapscript-code - or is
> MapServer finding the qix-file itself? 
> Is it possible to test if the qix-file really is used?
> 
> 2. Use minscale/maxscale to show less detailed maps when
> showing the complete layer. The maps could be created in (at
> least) three ways:
>       - thin out data with ArcView tools or another tool?
>       - use a raster "screenshot" created with shp2img.
>       - use "postgis and download the simplify function that
> implements the DP-algorithm 
> (http://cobalt.rmnet.it/~san/postgis/simplify/simplify-0.2.t
> gz."
> JC: 
> It is only the large files that should be thinned out - I am
> not interested in small files getting lesser detailed. It is
> maybe possible to check the filesize and only do the
> "thinning" and minscale/maxscale on large files. Howewer the
> thinning (or screenshot-making) should be done in runtime
> because the data is updated regularly. 
> Is this possible?
> Does anyone know the tool to thin out shape-files, that
> Arnulf mentioned in his second mail? (I haven't been looking
> for it myself yet, but just in case someone have it right by
> ....)
> 
> 3.  Throw the shapes into PG/PostGIS.  
> JC:  
> Is this the same solution as nr 2.3?
> 
> 4. "Separate the MapServer software and the PostGreSQL DB to
> run on separate machines
>  (hardware)". 
> JC: 
> Could be a idea later on
> 
> 5. Convert MapInfo files to shapefiles (and then use
> shapetree etc.)
> JC: 
> This should also be done in runtime (first time the file is
> read) - and then followed by shapetree etc. from above. 
> How to do it anyway? 
> 
> All in all I must admit: It seems a lot more triggy to deal
> with my speed-problems than I expected. 
> If anyone have other ideas I'm still all ears... (could you
> say that in english?)
> 
> cheers
> 
> Jan Christiansen
> 
>  
> > On 9 May 2003 at 13:14, jankarin at post1.dknet.dk wrote:
> > 
> > > Hello
> > > 
> > > I have a little problem with the speed in MapServer. 
> > > We're trying to make qiuck-viewer to shape- and
> > tab-formats, but 
> > > MapServer is running slow with large data: 20 seconds:
> > Shapefil 17 MB,
> > > dbf-fil 9,5 MB. 45 seconds: Map-fil 31 MB, Dat-fil 25
> MB.
> > > 
> > > I have been trying with the GCI-version and PHP
> Mapscript
> > - same
> > > result.
> > > 
> > > I am running MapServer 3.6.4 with PNG-output under
> Apache
> > 2.4.5 on
> > > Windows 2000. The pc is Pentim III, 650 Mhz, 262 MB
> RAM.
> > > 
> > > In another installation (PII, 400 Mhz, 196 MB RAM) a
> > shapefile
> > > (shapefile 17 MB, dbf-fil 9,5 MB) took 80 seconds to
> > show.
> > > 
> > > We're using MapServer in an application which is
> > dynamically creating
> > > map-files with Mapscript (only showing one layer at a
> > time). We're
> > > talknig about 17.000 layers in shape- and tabformats,
> so
> > it is not
> > > possible to improve indexfiles or anything else related
> to
> > a single
> > > layer.
> > > 
> > > Can anyone help?
> > > 
> > > Thanks,
> > > 
> > > Jan Christiansen
> > > jankarin at post1.dknet.dk
> > > Engineering College of Copenhagen
> > > 
> > > _______________________________________________
> > > Mapserver-users mailing list
> > > Mapserver-users at lists.gis.umn.edu
> > >
> http://lists.gis.umn.edu/mailman/listinfo/mapserver-users
> > > 
> > 
> > 
> 
> 
> _______________________________________________
> Mapserver-users mailing list
> Mapserver-users at lists.gis.umn.edu
> http://lists.gis.umn.edu/mailman/listinfo/mapserver-users