[mapserver-users] MapServer consuming 100% CPU
ritesh ambastha
ritesh.linux at gmail.com
Mon Feb 25 09:39:53 EST 2008
Hey,
I am confident about PostGIS which can solve your problem. Do give it a try.
Regards,
Ritesh Ambastha
On 2/25/08, Desarrollo Desarrollo <devmapserver at gmail.com> wrote:
>
> Hello Bart!
> Following your advice, I have installed MySQL and downloaded one public
> access shapefile, to let you (and everybody who wants to try this test)
> reproduce the results
>
>
> Download it from
> http://edcftp.cr.usgs.gov/pub/data/nationalatlas/hydrogm020.tar.gz that
> have 2 shapefiles : lines (rivers) and polygons (lakes). I'm going to use
> hydrogl020.shp :
>
> D:\ms4w\apps\maps\data\test\hydrogm020>ogrinfo -so -al hydrogl020.shp
> INFO: Open of `hydrogl020.shp'
> using driver `ESRI Shapefile' successful.
>
> Layer name: hydrogl020
> Geometry: Line String
> Feature Count: 76975
> Extent: (-179.998245, 17.674692) - (179.983139, 71.398186)
> Layer SRS WKT:
> (unknown)
> FNODE_: Integer (9.0)
> TNODE_: Integer (9.0)
> LPOLY_: Integer (9.0)
> RPOLY_: Integer (9.0)
> LENGTH: Real (13.11)
> HYDROGM020: Integer (9.0)
> FEATURE: String (80.0)
> F_CODE: Integer (2.0)
> NAME: String (80.0)
> STATE: String (5.0)
> STATE_FIPS: String (5.0)
> NAME_UC: String (80.0)
>
> Let's create qix index:
> D:\ms4w\apps\maps\data\test\hydrogm020>shptree hydrogl020
> creating index of new LSB format
>
>
> I'm going to test the shapefile, then I will convert it to a MySQL table
> and I will do the tests again
>
>
> SHAPEFILE
> 3 tests, let's use shp2img that skips apache, http requests, etc. Only one
> request in the command line.
>
> a1) No data access
> a2) Data access, only used in expressions to change the line color, no
> labeling
> a3) Data access only used for labeling
>
>
>
> ---------------------------
> --------- TEST A1 ---------
> ---------------------------
> Shapefile Map without data access
>
> MAP
> NAME "US RIVERS"
> SIZE 800 600
> IMAGETYPE PNG
> EXTENT -179.998245 17.674692 179.983139 71.398186
>
>
> PROJECTION
> "init=epsg:4269"
> END
>
> LAYER
> NAME "hydrogl020"
> STATUS OFF
> TYPE LINE
> DATA "hydrogl020"
> CLASS
> STYLE
> COLOR 0 0 0
> END
> END
> END
> END
>
> Result Test A1:
> --------------
> D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
> d:\ms4w\apps\maps\data\test\hydrogm020\a1.map -o
> D:\ms4w\apps\maps\data\test\hydrogm020\a1.png -e -179.998245 17.674692 -
> 60.0 71.398186 -t -l hydrogl020 -i PNG -all_debug 2
> msDrawMap(): Layer 0 (hydrogl020), 1.328s
> msDrawMap(): Drawing Label Cache, 0.000s
> msDrawMap() total time: 1.328s
>
>
>
> ---------------------------
> --------- TEST A2 ---------
> ---------------------------
> Shapefile Map with data access in expression
>
> MAP
> NAME "US RIVERS"
> SIZE 800 600
> IMAGETYPE PNG
> EXTENT -179.998245 17.674692 179.983139 71.398186
>
>
> PROJECTION
> "init=epsg:4269"
> END
>
> LAYER
> NAME "hydrogl020"
> STATUS OFF
> TYPE LINE
> DATA "hydrogl020"
> CLASS
> EXPRESSION([F_CODE] = 4)
> STYLE
> COLOR 0 0 0
> END
> END
> CLASS
> STYLE
> COLOR 255 0 0
> END
> END
> END
> END
>
>
> Result Test A2:
> ---------------
> D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
> d:\ms4w\apps\maps\data\test\hydrogm020\a2.map -o
> D:\ms4w\apps\maps\data\test\hydrogm020\a2.png -e -179.998245 17.674692 -
> 60.0 71.398186 -t -l hydrogl020 -i PNG -all_debug 2
> msDrawMap(): Layer 0 (hydrogl020), 6.250s
> msDrawMap(): Drawing Label Cache, 0.000s
> msDrawMap() total time: 6.250s
>
>
> ---------------------------
> --------- TEST A3 ---------
> ---------------------------
> Shapefile Map with data access
>
>
> MAP
> NAME "US RIVERS"
> SIZE 800 600
> IMAGETYPE PNG
> EXTENT -179.998245 17.674692 179.983139 71.398186
>
>
> PROJECTION
> "init=epsg:4269"
> END
>
> LAYER
> NAME "hydrogl020"
> STATUS OFF
> TYPE LINE
> DATA "hydrogl020"
> LABELITEM "STATE"
> CLASS
> STYLE
> COLOR 0 0 0
> END
> LABEL
> COLOR 51 51 51
> END
> END
> END
> END
>
> Result Test A3:
> --------------
> D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
> d:\ms4w\apps\maps\data\test\hydrogm020\a3.map -o
> D:\ms4w\apps\maps\data\test\hydrogm020\a3.png -e -179.998245 17.674692 -
> 60.0 71.398186 -t -l hydrogl020 -i PNG -all_debug 2
> msDrawMap(): Layer 0 (hydrogl020), 83.203s
> msDrawMap(): Drawing Label Cache, 0.703s
> msDrawMap() total time: 83.906s
>
>
>
> Ok, a2 shows that using F_CODE to determine the color of the line is not
> really very hard to mapserver but a3 shows that labeling is very hard to
> MapServer.
> It's time to repeat the tests with MySQL Spatial Data.
> Export shapefile to MySQL:
> ----------------
> D:\ms4w\apps\maps\data\test\hydrogm020>ogr2ogr -f MySQL
> MySQL:maps,host=localhost,user=mapserver,password=mapserver hydrogl020.shp-nln hydrogl020 -update -overwrite -lco engine=MYISAM
>
>
> Repeat the 3 tests
>
> ---------------------------
> --------- TEST B1 ---------
> ---------------------------
> OGR MySQL Map without data access
>
>
> MAP
> NAME "US RIVERS"
> SIZE 800 600
> IMAGETYPE PNG
> EXTENT -179.998245 17.674692 179.983139 71.398186
>
>
> PROJECTION
> "init=epsg:4269"
> END
>
> LAYER
> NAME "hydrogl020"
> STATUS OFF
> TYPE LINE
> CONNECTIONTYPE OGR
> CONNECTION
> "MySQL:maps,host=localhost,user=mapserver,password=mapserver"
> DATA "hydrogl020"
> CLASS
> STYLE
> COLOR 0 0 0
> END
> END
> END
> END
>
>
> Result Test B1:
> ---------------
> D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
> d:\ms4w\apps\maps\data\test\hydrogm020\b1.map -o
> D:\ms4w\apps\maps\data\test\hydrogm020\b1.png -e -179.998245 17.674692 -
> 60.0 71.398186 -t -l hydrogl020 -i PNG -all_debug 2
>
> msOGRFileOpen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)...
> OGROPen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)
>
> msOGRLayerClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver).
>
> msOGRFileClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver,1).
> msDrawMap(): Layer 0 (hydrogl020), 4.172s
> msDrawMap(): Drawing Label Cache, 0.000s
> msDrawMap() total time: 4.188s
>
>
> ---------------------------
> --------- TEST B2 ---------
> ---------------------------
> OGR MySQL Map with data access in expression
>
> MAP
> NAME "US RIVERS"
> SIZE 800 600
> IMAGETYPE PNG
> EXTENT -179.998245 17.674692 179.983139 71.398186
>
>
> PROJECTION
> "init=epsg:4269"
> END
>
> LAYER
> NAME "hydrogl020"
> STATUS OFF
> TYPE LINE
> CONNECTIONTYPE OGR
> CONNECTION
> "MySQL:maps,host=localhost,user=mapserver,password=mapserver"
> DATA "hydrogl020"
> CLASS
> EXPRESSION([F_CODE] = 4)
> STYLE
> COLOR 0 0 0
> END
> END
> CLASS
> STYLE
> COLOR 255 0 0
> END
> END
> END
> END
>
>
> Result Test B2:
> ---------------
> D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
> d:\ms4w\apps\maps\data\test\hydrogm020\b2.map -o
> D:\ms4w\apps\maps\data\test\hydrogm020\b2.png -e -179.998245 17.674692 -
> 60.0 71.398186 -t -l hydrogl020 -i PNG -all_debug 2
>
> msOGRFileOpen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)...
> OGROPen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)
>
> msOGRLayerClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver).
>
> msOGRFileClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver,1).
> msDrawMap(): Layer 0 (hydrogl020), 4.766s
> msDrawMap(): Drawing Label Cache, 0.000s
> msDrawMap() total time: 4.766s
>
>
>
> ---------------------------
> --------- TEST B3 ---------
> ---------------------------
> OGR MySQL Map with data access for labeling
>
> MAP
> NAME "US RIVERS"
> SIZE 800 600
> IMAGETYPE PNG
> EXTENT -179.998245 17.674692 179.983139 71.398186
>
>
> PROJECTION
> "init=epsg:4269"
> END
>
> LAYER
> NAME "hydrogl020"
> STATUS OFF
> TYPE LINE
> CONNECTIONTYPE OGR
> CONNECTION
> "MySQL:maps,host=localhost,user=mapserver,password=mapserver"
> DATA "hydrogl020"
> LABELITEM "STATE"
> CLASS
> STYLE
> COLOR 0 0 0
> END
> LABEL
> COLOR 51 51 51
> END
> END
> END
> END
>
>
> Result Test B3:
> ---------------
> D:\ms4w\apps\maps\data\test\hydrogm020>shp2img -m
> d:\ms4w\apps\maps\data\test\hydrogm020\b3.map -o
> D:\ms4w\apps\maps\data\test\hydrogm020\b3.png -e -179.998245 17.674692 -
> 60.0 71.398186 -t -l hydrogl020 -i PNG -all_debug 2
>
> msOGRFileOpen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)...
> OGROPen(MySQL:maps,host=localhost,user=mapserver,password=mapserver)
>
> msOGRLayerClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver).
>
> msOGRFileClose(MySQL:maps,host=localhost,user=mapserver,password=mapserver,1).
> msDrawMap(): Layer 0 (hydrogl020), 83.361s
> msDrawMap(): Drawing Label Cache, 0.781s
> msDrawMap() total time: 84.142s
>
>
>
> So, I think that MySQL doesn't solve the question, it seems to be a
> labeling issue. Could it be so hard to render the labels? I repeated the A3
> and B3 tests including FORCE ON in the LABEL object
> to avoid MapServer to calculate anything about lengths, the result was the
> same. How is it possible?
> I don't know if this is normal MapServer behavior and the cost of render
> labels is so high!
>
> Thank you very much in advance for your answers
> Sergio
>
>
>
>
> 2008/2/22, Desarrollo Desarrollo <devmapserver at gmail.com>:
> >
> > Thank you very much for your answer,
> >
> > Therefore, Should I think that Shapefiles / DBF data is not suitable for
> > MapServer if the number of features in the shapefile is a "big" number like
> > 473815 line strings in the Streets.shp? I've tried with MajHwys.shp
> > (35295 linestrings features) labeling and using expressions and works
> > ok. Streets.shp with its 473815 linestrings is rendered fast if there is
> > not DBF access (not labeling neither expressions), but when DBF access is
> > needed Mapservers hangs up!
> > Will MySQL spatial data increase the speed and number of features that
> > MapServer can manage?
> >
> > Best regards
> >
> >
> > 2008/2/21, Bart van den Eijnden (OSGIS) <bartvde at osgis.nl>:
> > >
> > > It is because Mapserver cannot use any attribute indexes on the DBF.
> > >
> > > Put your shapefile in a spatial database like PostGIS I would suggest
> > > and
> > > create attribute indexes.
> > >
> > > Best regards,
> > > Bart
> > >
> > > --
> > > Bart van den Eijnden
> > > OSGIS, Open Source GIS
> > > http://www.osgis.nl
> > >
> > >
> > > --------- Oorspronkelijk bericht --------
> > > Van: Paul Spencer <pspencer at dmsolutions.ca>
> > > Naar: Desarrollo Desarrollo <devmapserver at gmail.com>
> > > Cc: mapserver-users at lists.osgeo.org
> > > Onderwerp: Re: [mapserver-users] MapServer consuming 100% CPU
> > > Datum: 21/02/08 11:09
> > >
> > >
> > > > See:
> > > >
> > > > http://mapserver.gis.umn.edu/docs/howto/mapfiletuning
> > > > http://mapserver.gis.umn.edu/docs/howto/optimizevector
> > > > http://mapserver.gis.umn.edu/docs/howto/tileindex
> > > >
> > > > Cheers
> > > >
> > > > Paul
> > > >
> > > > On 21-Feb-08, at 2:56 AM, Desarrollo Desarrollo wrote:
> > > >
> > >
> > > > > Hi,
> > > > >
> > > > > I am developing an application with MapServer v2.2.7 with
> > > > > Apache on Windows XP, but I have a serious problem, below I
> > > > > describe it:
> > > > >
> > > > > Using Navteq demo maps from Paris, the Streets.shp have around
> > > > > 500000 line strings features, when mapserver renders the layer
> > > > > without using DBF data (only draw lines) everything is OK and
> > > fast,
> > >
> > > > > but when mapserver needs to access to the DBF to render labels
> > > or
> > > > > use expressions to filter data, mapserver starts to consuming
> > > 100%
> > > > > CPU and nothing is rendered after to spends a long time.
> > > > >
> > > > > Using Teleatlas demo maps for a medium city of Spain, the
> > > Street.shp
> > >
> > > > > with around 11000 line strings features doesn't show any
> > > trouble and
> > >
> > > > > mapserver renders the images OK and fast even accessing to DBF
> > > to
> > > > > render labels
> > > > >
> > > > > Could somebody help me with this problem? Why does it happen?
> > > How do
> > >
> > > > > I solve it?
> > > > >
> > > > >
> > > > > Thank you very much.
> > > > > _______________________________________________
> > > > > mapserver-users mailing list
> > > > > mapserver-users at lists.osgeo.org
> > > > > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> > >
> > > >
> > > >
> > > > __________________________________________
> > > >
> > > > Paul Spencer
> > > > Chief Technology Officer
> > > > DM Solutions Group Inc
> > > > http://www.dmsolutions.ca/
> > > >
> > > > _______________________________________________
> > > > mapserver-users mailing list
> > > > mapserver-users at lists.osgeo.org
> > > > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
>
--
Ritesh Ambastha,
Project Manager
Mobiance Technologies,
Bangalore
+91-80-41264755
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/mapserver-users/attachments/20080225/b93c5683/attachment-0001.html
More information about the mapserver-users
mailing list