[mapserver-users] MapServer consuming 100% CPU
Desarrollo Desarrollo
devmapserver at gmail.com
Mon Feb 25 08:48:35 EST 2008
Hello Bart!
Following your advice, I have installed MySQL and downloaded one public
access shapefile, to let you (and everybody who wants to try this test)
reproduce the results
Download it from
http://edcftp.cr.usgs.gov/pub/data/nationalatlas/hydrogm020.tar.gz that have
2 shapefiles : lines (rivers) and polygons (lakes). I'm going to use
hydrogl020.shp :
D:\ms4w\apps\maps\data\test\hydrogm020>ogrinfo -so -al hydrogl020.shp
INFO: Open of `hydrogl020.shp'
using driver `ESRI Shapefile' successful.
Layer name: hydrogl020
Geometry: Line String
Feature Count: 76975
Extent: (-179.998245, 17.674692) - (179.983139, 71.398186)
Layer SRS WKT:
(unknown)
FNODE_: Integer (9.0)
TNODE_: Integer (9.0)
LPOLY_: Integer (9.0)
RPOLY_: Integer (9.0)
LENGTH: Real (13.11)
HYDROGM020: Integer (9.0)
FEATURE: String (80.0)
F_CODE: Integer (2.0)
NAME: String (80.0)
STATE: String (5.0)
STATE_FIPS: String (5.0)
NAME_UC: String (80.0)
Let's create qix index:
D:\ms4w\apps\maps\data\test\hydrogm020>shptree hydrogl020
creating index of new LSB format
I'm going to test the shapefile, then I will convert it to a MySQL table and
I will do the tests again
SHAPEFILE
3 tests, let's use shp2img that skips apache, http requests, etc. Only one
request in the command line.
a1) No data access
a2) Data access, only used in expressions to change the line color, no
labeling
a3) Data access only used for labeling
---------------------------
--------- TEST A1 ---------
---------------------------
Shapefile Map without data access
MAP
NAME "US RIVERS"
SIZE 800 600
IMAGETYPE PNG
EXTENT -179.998245 17.674692 179.983139 71.398186
PROJECTION
"init=epsg:4269"
END
LAYER
NAME "hydrogl020"
STATUS OFF
TYPE LINE
DATA "hydrogl020"
CLASS
STYLE
COLOR 0 0 0
END
END
END
END
Result Test A1:
--------------
D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
d:\ms4w\apps\maps\data\test\hydrogm020\a1.map -o
D:\ms4w\apps\maps\data\test\hydrogm020\a1.png -e -179.998245 17.674692 -60.0
71.398186 -t -l hydrogl020 -i PNG -all_debug 2
msDrawMap(): Layer 0 (hydrogl020), 1.328s
msDrawMap(): Drawing Label Cache, 0.000s
msDrawMap() total time: 1.328s
---------------------------
--------- TEST A2 ---------
---------------------------
Shapefile Map with data access in expression
MAP
NAME "US RIVERS"
SIZE 800 600
IMAGETYPE PNG
EXTENT -179.998245 17.674692 179.983139 71.398186
PROJECTION
"init=epsg:4269"
END
LAYER
NAME "hydrogl020"
STATUS OFF
TYPE LINE
DATA "hydrogl020"
CLASS
EXPRESSION([F_CODE] = 4)
STYLE
COLOR 0 0 0
END
END
CLASS
STYLE
COLOR 255 0 0
END
END
END
END
Result Test A2:
---------------
D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
d:\ms4w\apps\maps\data\test\hydrogm020\a2.map -o
D:\ms4w\apps\maps\data\test\hydrogm020\a2.png -e -179.998245 17.674692 -60.0
71.398186 -t -l hydrogl020 -i PNG -all_debug 2
msDrawMap(): Layer 0 (hydrogl020), 6.250s
msDrawMap(): Drawing Label Cache, 0.000s
msDrawMap() total time: 6.250s
---------------------------
--------- TEST A3 ---------
---------------------------
Shapefile Map with data access
MAP
NAME "US RIVERS"
SIZE 800 600
IMAGETYPE PNG
EXTENT -179.998245 17.674692 179.983139 71.398186
PROJECTION
"init=epsg:4269"
END
LAYER
NAME "hydrogl020"
STATUS OFF
TYPE LINE
DATA "hydrogl020"
LABELITEM "STATE"
CLASS
STYLE
COLOR 0 0 0
END
LABEL
COLOR 51 51 51
END
END
END
END
Result Test A3:
--------------
D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
d:\ms4w\apps\maps\data\test\hydrogm020\a3.map -o
D:\ms4w\apps\maps\data\test\hydrogm020\a3.png -e -179.998245 17.674692 -60.0
71.398186 -t -l hydrogl020 -i PNG -all_debug 2
msDrawMap(): Layer 0 (hydrogl020), 83.203s
msDrawMap(): Drawing Label Cache, 0.703s
msDrawMap() total time: 83.906s
Ok, a2 shows that using F_CODE to determine the color of the line is not
really very hard to mapserver but a3 shows that labeling is very hard to
MapServer.
It's time to repeat the tests with MySQL Spatial Data.
Export shapefile to MySQL:
----------------
D:\ms4w\apps\maps\data\test\hydrogm020>ogr2ogr -f MySQL
MySQL:maps,host=localhost,user=mapserver,password=mapserver
hydrogl020.shp-nln hydrogl020 -update -overwrite -lco engine=MYISAM
Repeat the 3 tests
---------------------------
--------- TEST B1 ---------
---------------------------
OGR MySQL Map without data access
MAP
NAME "US RIVERS"
SIZE 800 600
IMAGETYPE PNG
EXTENT -179.998245 17.674692 179.983139 71.398186
PROJECTION
"init=epsg:4269"
END
LAYER
NAME "hydrogl020"
STATUS OFF
TYPE LINE
CONNECTIONTYPE OGR
CONNECTION
"MySQL:maps,host=localhost,user=mapserver,password=mapserver"
DATA "hydrogl020"
CLASS
STYLE
COLOR 0 0 0
END
END
END
END
Result Test B1:
---------------
D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
d:\ms4w\apps\maps\data\test\hydrogm020\b1.map -o
D:\ms4w\apps\maps\data\test\hydrogm020\b1.png -e -179.998245 17.674692 -60.0
71.398186 -t -l hydrogl020 -i PNG -all_debug 2
msOGRFileOpen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)...
OGROPen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)
msOGRLayerClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver).
msOGRFileClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver,1).
msDrawMap(): Layer 0 (hydrogl020), 4.172s
msDrawMap(): Drawing Label Cache, 0.000s
msDrawMap() total time: 4.188s
---------------------------
--------- TEST B2 ---------
---------------------------
OGR MySQL Map with data access in expression
MAP
NAME "US RIVERS"
SIZE 800 600
IMAGETYPE PNG
EXTENT -179.998245 17.674692 179.983139 71.398186
PROJECTION
"init=epsg:4269"
END
LAYER
NAME "hydrogl020"
STATUS OFF
TYPE LINE
CONNECTIONTYPE OGR
CONNECTION
"MySQL:maps,host=localhost,user=mapserver,password=mapserver"
DATA "hydrogl020"
CLASS
EXPRESSION([F_CODE] = 4)
STYLE
COLOR 0 0 0
END
END
CLASS
STYLE
COLOR 255 0 0
END
END
END
END
Result Test B2:
---------------
D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
d:\ms4w\apps\maps\data\test\hydrogm020\b2.map -o
D:\ms4w\apps\maps\data\test\hydrogm020\b2.png -e -179.998245 17.674692 -60.0
71.398186 -t -l hydrogl020 -i PNG -all_debug 2
msOGRFileOpen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)...
OGROPen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)
msOGRLayerClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver).
msOGRFileClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver,1).
msDrawMap(): Layer 0 (hydrogl020), 4.766s
msDrawMap(): Drawing Label Cache, 0.000s
msDrawMap() total time: 4.766s
---------------------------
--------- TEST B3 ---------
---------------------------
OGR MySQL Map with data access for labeling
MAP
NAME "US RIVERS"
SIZE 800 600
IMAGETYPE PNG
EXTENT -179.998245 17.674692 179.983139 71.398186
PROJECTION
"init=epsg:4269"
END
LAYER
NAME "hydrogl020"
STATUS OFF
TYPE LINE
CONNECTIONTYPE OGR
CONNECTION
"MySQL:maps,host=localhost,user=mapserver,password=mapserver"
DATA "hydrogl020"
LABELITEM "STATE"
CLASS
STYLE
COLOR 0 0 0
END
LABEL
COLOR 51 51 51
END
END
END
END
Result Test B3:
---------------
D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
d:\ms4w\apps\maps\data\test\hydrogm020\b3.map -o
D:\ms4w\apps\maps\data\test\hydrogm020\b3.png -e -179.998245 17.674692 -60.0
71.398186 -t -l hydrogl020 -i PNG -all_debug 2
msOGRFileOpen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)...
OGROPen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)
msOGRLayerClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver).
msOGRFileClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver,1).
msDrawMap(): Layer 0 (hydrogl020), 83.361s
msDrawMap(): Drawing Label Cache, 0.781s
msDrawMap() total time: 84.142s
So, I think that MySQL doesn't solve the question, it seems to be a labeling
issue. Could it be so hard to render the labels? I repeated the A3 and B3
tests including FORCE ON in the LABEL object
to avoid MapServer to calculate anything about lengths, the result was the
same. How is it possible?
I don't know if this is normal MapServer behavior and the cost of render
labels is so high!
Thank you very much in advance for your answers
Sergio
2008/2/22, Desarrollo Desarrollo <devmapserver at gmail.com>:
>
> Thank you very much for your answer,
>
> Therefore, Should I think that Shapefiles / DBF data is not suitable for
> MapServer if the number of features in the shapefile is a "big" number like
> 473815 line strings in the Streets.shp? I've tried with MajHwys.shp
> (35295 linestrings features) labeling and using expressions and works ok.
> Streets.shp with its 473815 linestrings is rendered fast if there is not
> DBF access (not labeling neither expressions), but when DBF access is needed
> Mapservers hangs up!
> Will MySQL spatial data increase the speed and number of features that
> MapServer can manage?
>
> Best regards
>
>
> 2008/2/21, Bart van den Eijnden (OSGIS) <bartvde at osgis.nl>:
> >
> > It is because Mapserver cannot use any attribute indexes on the DBF.
> >
> > Put your shapefile in a spatial database like PostGIS I would suggest
> > and
> > create attribute indexes.
> >
> > Best regards,
> > Bart
> >
> > --
> > Bart van den Eijnden
> > OSGIS, Open Source GIS
> > http://www.osgis.nl
> >
> >
> > --------- Oorspronkelijk bericht --------
> > Van: Paul Spencer <pspencer at dmsolutions.ca>
> > Naar: Desarrollo Desarrollo <devmapserver at gmail.com>
> > Cc: mapserver-users at lists.osgeo.org
> > Onderwerp: Re: [mapserver-users] MapServer consuming 100% CPU
> > Datum: 21/02/08 11:09
> >
> >
> > > See:
> > >
> > > http://mapserver.gis.umn.edu/docs/howto/mapfiletuning
> > > http://mapserver.gis.umn.edu/docs/howto/optimizevector
> > > http://mapserver.gis.umn.edu/docs/howto/tileindex
> > >
> > > Cheers
> > >
> > > Paul
> > >
> > > On 21-Feb-08, at 2:56 AM, Desarrollo Desarrollo wrote:
> > >
> >
> > > > Hi,
> > > >
> > > > I am developing an application with MapServer v2.2.7 with
> > > > Apache on Windows XP, but I have a serious problem, below I
> > > > describe it:
> > > >
> > > > Using Navteq demo maps from Paris, the Streets.shp have around
> > > > 500000 line strings features, when mapserver renders the layer
> > > > without using DBF data (only draw lines) everything is OK and
> > fast,
> >
> > > > but when mapserver needs to access to the DBF to render labels or
> > > > use expressions to filter data, mapserver starts to consuming
> > 100%
> > > > CPU and nothing is rendered after to spends a long time.
> > > >
> > > > Using Teleatlas demo maps for a medium city of Spain, the
> > Street.shp
> >
> > > > with around 11000 line strings features doesn't show any trouble
> > and
> >
> > > > mapserver renders the images OK and fast even accessing to DBF
> > to
> > > > render labels
> > > >
> > > > Could somebody help me with this problem? Why does it happen? How
> > do
> >
> > > > I solve it?
> > > >
> > > >
> > > > Thank you very much.
> > > > _______________________________________________
> > > > mapserver-users mailing list
> > > > mapserver-users at lists.osgeo.org
> > > > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> >
> > >
> > >
> > > __________________________________________
> > >
> > > Paul Spencer
> > > Chief Technology Officer
> > > DM Solutions Group Inc
> > > http://www.dmsolutions.ca/
> > >
> > > _______________________________________________
> > > mapserver-users mailing list
> > > mapserver-users at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> > >
> > >
> >
> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/mapserver-users/attachments/20080225/fc3a366f/attachment-0001.html
More information about the mapserver-users
mailing list