[mapserver-users] logging GDAL queries
Sebastian E. Ovide
sebastian.ovide at gmail.com
Fri Aug 13 10:27:31 EDT 2010
On Fri, Aug 13, 2010 at 1:59 PM, Ivan Lucena <ivan.lucena at pmldnet.com>wrote:
> Sebastian,
>
> You are using a very small block size, maybe because you are also using
> ArcGIS products and they only support Oracle GeoRaster with 256x256
> blocking. Is that right?
>
nop... I guess that 256x256 if the default gdal_translate block size ?
>
> Because of that, to access your GeoRaster object, the GDAL driver is
> requesting a very large array of BLOB (1000000 approximately). That can make
> your sever such-up memory from the machine where it is running and it is
> probably using virtual memory (disk) what is really slow.
do you mean where MapServer is running ?
top - 14:41:31 up 7 days, 6:20, 10 users, load average: 2.81, 1.92, 1.20
Tasks: 340 total, 2 running, 338 sleeping, 0 stopped, 0 zombie
Cpu(s): 11.8%us, 10.4%sy, 7.7%ni, 62.1%id, 0.0%wa, 0.0%hi, 8.0%si,
0.0%st
Mem: 12331240k total, 5992136k used, 6339104k free, 189416k buffers
Swap: 9084716k total, 703708k used, 8381008k free, 1481992k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24850 www-data 20 0 533m 330m 18m R 14 2.7 0:00.94 mapserv
24852 www-data 20 0 532m 329m 18m S 19 2.7 0:01.22 mapserv
24849 www-data 20 0 529m 326m 18m S 16 2.7 0:01.00 mapserv
24851 www-data 20 0 517m 314m 18m S 13 2.6 0:00.96 mapserv
24853 www-data 20 0 499m 296m 18m S 8 2.5 0:00.75 mapserv
24854 www-data 20 0 470m 267m 18m S 5 2.2 0:00.53 mapserv
> It also uses some memory on the client side just to hold the array too but
> if you are running everything (Mapserver,Oracle) on the same machine that
> will adds up.
>
Oracle is running on another Machine... and is very likely that it is the
bottle neck...
> We are aware of that problem but it would be very helpful if you could file
> a bug report on GDAL's trac [http://trac.osgeo.org/gdal/] Thanks.
>
>
I guess that the problem is that MapServer triys to read the whole map
instead of reading only tile ???? is that correct ?
> Just as curiosity. Did you loaded that image with gdal_translate? How long
> it took and long does it takes to generate pyramids? Ha, you forgot to say
> what version of Oracle you are using.
>
>
I loaded it with gdal in around 5 minutes... the pyramid (all levels) has
been created overnight....
> Best regards,
>
> Ivan
>
>
>
> > -------Original Message-------
> > From: Sebastian E. Ovide <sebastian.ovide at gmail.com>
> > To: Lucena, Ivan <ivan.lucena at pmldnet.com>
> > Cc: mapserver-users at lists.osgeo.org
> > Subject: Re: [mapserver-users] logging GDAL queries
> > Sent: Aug 13 '10 06:44
> >
> > sure...
> >
> >
> > sebas at SeansPC:~/mapserver/mapserver-5.6.5$ gdalinfo --version
> > GDAL 1.7.2, released 2010/04/23
> >
> > sebas at SeansPC:~/mapserver/mapserver-5.6.5$ gdalinfo
> > georaster:geoserver,geoserver,MFPRODUK_11G,fluvd04q200pj,georaster,id=1
> > -mdd oracle
> > Driver: GeoRaster/Oracle Spatial GeoRaster
> > Files: none associated
> > Size is 190325, 271423
> > Coordinate System is:
> > PROJCS["Popular Visualisation CRS / Mercator (deprecated)",
> > GEOGCS["Popular Visualisation CRS",
> > DATUM["Popular_Visualisation_Datum",
> > SPHEROID["Popular Visualisation Sphere",6378137,0,
> > AUTHORITY["EPSG","7059"]],
> > TOWGS84[0,0,0,0,0,0,0],
> > AUTHORITY["EPSG","6055"]],
> > PRIMEM["Greenwich",0,
> > AUTHORITY["EPSG","8901"]],
> > UNIT["degree",0.01745329251994328,
> > AUTHORITY["EPSG","9122"]],
> > AUTHORITY["EPSG","4055"]],
> > UNIT["metre",1,
> > AUTHORITY["EPSG","9001"]],
> > PROJECTION["Mercator_1SP"],
> > PARAMETER["central_meridian",0],
> > PARAMETER["scale_factor",1],
> > PARAMETER["false_easting",0],
> > PARAMETER["false_northing",0],
> > EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0
> > +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext
> > +no_defs"],
> > AUTHORITY["EPSG","3785"],
> > AXIS["X",EAST],
> > AXIS["Y",NORTH]]
> > Origin = (-703633.464883987908252,7805920.616936270147562)
> > Pixel Size = (4.999999999999999,-5.000000000000000)
> > Metadata (oracle):
> > TABLE_NAME=fluvd04q200pj
> > COLUMN_NAME=georaster
> > RDT_TABLE_NAME=RDT_FLUVD04Q200PJ
> > RASTER_ID=62
> > METADATA=<georasterMetadata xmlns="[LINK:
> > http://xmlns.oracle.com/spatial/georaster]
> > http://xmlns.oracle.com/spatial/georaster">
> > <objectInfo>
> > <rasterType>20001</rasterType>
> > <isBlank>false</isBlank>
> > <defaultRed>1</defaultRed>
> > <defaultGreen>1</defaultGreen>
> > <defaultBlue>1</defaultBlue>
> > </objectInfo>
> > <rasterInfo>
> > <cellRepresentation>UNDEFINED</cellRepresentation>
> > <cellDepth>8BIT_U</cellDepth>
> > <NODATA>255</NODATA>
> > <totalDimensions>2</totalDimensions>
> > <dimensionSize type="ROW">
> > <size>271423</size>
> > </dimensionSize>
> > <dimensionSize type="COLUMN">
> > <size>190325</size>
> > </dimensionSize>
> > <ULTCoordinate>
> > <row>0</row>
> > <column>0</column>
> > </ULTCoordinate>
> > <blocking>
> > <type>REGULAR</type>
> > <totalRowBlocks>1061</totalRowBlocks>
> > <totalColumnBlocks>744</totalColumnBlocks>
> > <rowBlockSize>256</rowBlockSize>
> > <columnBlockSize>256</columnBlockSize>
> > </blocking>
> > <interleaving>BIP</interleaving>
> > <pyramid>
> > <type>DECREASE</type>
> > <resampling>NN</resampling>
> > <maxLevel>11</maxLevel>
> > </pyramid>
> > <compression>
> > <type>DEFLATE</type>
> > </compression>
> > </rasterInfo>
> > <spatialReferenceInfo>
> > <isReferenced>true</isReferenced>
> > <isRectified>true</isRectified>
> > <SRID>3785</SRID>
> > <spatialResolution dimensionType="X">
> > <resolution>5</resolution>
> > </spatialResolution>
> > <spatialResolution dimensionType="Y">
> > <resolution>5</resolution>
> > </spatialResolution>
> > <modelCoordinateLocation>CENTER</modelCoordinateLocation>
> > <modelType>FunctionalFitting</modelType>
> > <polynomialModel rowOff="0" columnOff="0" xOff="0" yOff="0" zOff="0"
> > rowScale="1" columnScale="1" xScale="1" yScale="1" zScale="1">
> > <pPolynomial pType="1" nVars="2" order="1" nCoefficients="3">
> > <polynomialCoefficients>1561183.623387254 0
> > -0.2</polynomialCoefficients>
> > </pPolynomial>
> > <qPolynomial pType="1" nVars="0" order="0" nCoefficients="1">
> > <polynomialCoefficients>1</polynomialCoefficients>
> > </qPolynomial>
> > <rPolynomial pType="1" nVars="2" order="1" nCoefficients="3">
> > <polynomialCoefficients>140726.1929767976 0.2
> > 0</polynomialCoefficients>
> > </rPolynomial>
> > <sPolynomial pType="1" nVars="0" order="0" nCoefficients="1">
> > <polynomialCoefficients>1</polynomialCoefficients>
> > </sPolynomial>
> > </polynomialModel>
> > </spatialReferenceInfo>
> > <layerInfo>
> > <layerDimension>BAND</layerDimension>
> > <objectLayer>
> > <layerNumber>0</layerNumber>
> > <layerDimensionOrdinate>0</layerDimensionOrdinate>
> > <layerID/>
> > <colorMap>
> > <colors>
> > <cell value="1" red="255" green="0" blue="0"
> > alpha="100"/>
> > <cell value="2" red="0" green="255" blue="0"
> > alpha="100"/>
> > <cell value="3" red="0" green="0" blue="255"
> > alpha="100"/>
> > <cell value="4" red="150" green="50" blue="100"
> > alpha="100"/>
> > <cell value="255" red="0" green="0" blue="0" alpha="0"/>
> > </colors>
> > </colorMap>
> > </objectLayer>
> > </layerInfo>
> > </georasterMetadata>
> >
> > Image Structure Metadata:
> > INTERLEAVE=PIXEL
> > COMPRESSION=DEFLATE
> > Corner Coordinates:
> > Upper Left ( -703633.465, 7805920.617) ( 6d19'15.05"W, 57d
> > 2'52.80"N)
> > Lower Left ( -703633.465, 6448805.617) ( 6d19'15.05"W,
> > 49d49'30.04"N)
> > Upper Right ( 247991.535, 7805920.617) ( 2d13'39.89"E, 57d
> > 2'52.80"N)
> > Lower Right ( 247991.535, 6448805.617) ( 2d13'39.89"E,
> > 49d49'30.04"N)
> > Center ( -227820.965, 7127363.117) ( 2d 2'47.58"W,
> > 53d35'23.19"N)
> > Band 1 Block=256x256 Type=Byte, ColorInterp=Gray
> > NoData Value=255
> > Overviews: 95162x135711, 47581x67855, 23790x33927, 11895x16963,
> > 5947x8481, 2973x4240, 1486x2120, 743x1060, 371x530, 185x265, 92x132
> >
> >
> > On Fri, Aug 13, 2010 at 12:32 PM, Lucena, Ivan <[LINK:
> > mailto:ivan.lucena at pmldnet.com] ivan.lucena at pmldnet.com> wrote:
> > Interesting.
> > Can you post your gdalinfo report with "-mdd oracle" option?
> > What are the versions your are running (GDAL,Oracle)?
> > Older version of the driver used to run a new query every time you
> change
> > levels.
> > Sebastian E. Ovide wrote:
> >
> > yep I did... and the weir thing is that generating the images from "far
> > away" zoom under 12... it performs very well... not sure why it doesn't
> > in the levels 13-17.... for that reason I was trying to find the SQL
> > queries....
> >
> >
> > On Fri, Aug 13, 2010 at 12:01 PM, Lucena, Ivan <[LINK:
> > mailto:ivan.lucena at pmldnet.com] ivan.lucena at pmldnet.com <mailto:[LINK:
> > mailto:ivan.lucena at pmldnet.com] ivan.lucena at pmldnet.com>> wrote:
> >
> > Sebastian,
> >
> >
> > Sebastian E. Ovide wrote:
> >
> > Hi All
> >
> > I am using MapServer + GDAL to render GeoRasters and I am
> > having
> > very poor performance reading GeoRasters at a high zoom
> > level...
> > over 13... it would take even 20 seconds per tile...
> >
> >
> > Did you generate pyramids?
> >
> >
> > Is it possible to log the GDAL queries that MapServer is
> > doing?
> > (and the SQL queries that GDAL is doing ?)
> >
> > thanks
> >
> > -- Sebastian E. Ovide
> >
> >
> >
> >
> ------------------------------------------------------------------------
> >
> >
> > _______________________________________________
> > mapserver-users mailing list
> > [LINK: mailto:mapserver-users at lists.osgeo.org]
> > mapserver-users at lists.osgeo.org
> > <mailto:[LINK: mailto:mapserver-users at lists.osgeo.org]
> > mapserver-users at lists.osgeo.org>
> >
> > [LINK:
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users]
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> >
> >
> > --
> > Sebastian E. Ovide
> >
> >
> > --
> > Sebastian E. Ovide
>
--
Sebastian E. Ovide
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/mapserver-users/attachments/20100813/919530af/attachment-0001.html
More information about the mapserver-users
mailing list