[mapserver-dev] WFS + GEOS + native Oracle Spatial
+ <ogc:Intersects> = not working?
Vitali Diatchkov
vitali at arbonaut.com
Tue Mar 16 11:40:36 EDT 2010
Hi, Assefa.
You were right, a ticket 3305 is definitly about this issue. I have
recompiled latest codebase in branch 5-6 and the following filtering,
for example, started to work from UDIG:
====================
<GetFeature xmlns="http://www.opengis.net/wfs"
xmlns:gml="http://www.opengis.net/gml"
xmlns:ogc="http://www.opengis.net/ogc" version="1.0.0" service="WFS"
outputFormat="GML2"><Query handle="liteRenderer"
typeName="TAAKA_TIENOSA"><ogc:PropertyName>msGeometry</ogc:PropertyName>
<ogc:Filter><ogc:And><ogc:BBOX><ogc:PropertyName>msGeometry</ogc:PropertyName>
<gml:Box><gml:coordinates decimal="." cs="," ts="
">3378911.2334760367,7029122.276618615
3381287.88106233,7031323.68751652</gml:coordinates>
</gml:Box>
</ogc:BBOX>
<ogc:Intersects><ogc:PropertyName>msGeometry</ogc:PropertyName>
<gml:Polygon><gml:outerBoundaryIs><gml:LinearRing><gml:coordinates
decimal="." cs="," ts=" ">3380556.267888301,7030149.601704304
3380556.267888301,7030679.692686685 3380974.645481833,7030679.692686685
3380974.645481833,7030149.601704304
3380556.267888301,7030149.601704304</gml:coordinates>
</gml:LinearRing>
</gml:outerBoundaryIs>
</gml:Polygon>
</ogc:Intersects>
</ogc:And>
</ogc:Filter>
</Query>
</GetFeature>
=======================
But one issue (that was also faced earlier) is still in place. I am
using MapServer through FastCGI interface and seems in native
oraclespatial module there is a memory leak.
After MapServer is loaded into memory during first such a request (BBOX
+ Intersects) only 7-8 requests are handled (being sent from UDIG) and
MapServer ends up with the following error:
=======================
[Tue Mar 16 17:10:57 2010].590000 msOracleSpatialLayerWhichShapes():
OracleSpatial error. Cannot allocate layerinfo->items buffer
=======================
I suppose it's the case only for FastCGI mode while during one CGI
request the memory leak can not be caught.
the full log file for this request with DEBUG level 4 is:
========================
[Tue Mar 16 17:10:45 2010].263000 CGI Request 21 on process 7992
[Tue Mar 16 17:10:45 2010].263000 msOracleSpatialLayerOpen called with:
GEOLOC FROM TAAKA.TIENOSA USING UNIQUE TIENOSAID SRID 2393 (Layer
pointer 01E07A90)
[Tue Mar 16 17:10:45 2010].310000 msOracleSpatialLayerOpen. Shared
connection not available. Creating one.
[Tue Mar 16 17:10:45 2010].310000
msConnPoolRegister(TAAKA_TIENOSA,user/password at instance,0E12EFE0)
[Tue Mar 16 17:10:45 2010].310000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].310000 msOracleSpatialLayerGetItems was called.
[Tue Mar 16 17:10:45 2010].310000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].326000 msOracleSpatialLayerOpen called with:
GEOLOC FROM TAAKA.TIENOSA USING UNIQUE TIENOSAID SRID 2393 (Layer
pointer 01E07A90)
[Tue Mar 16 17:10:45 2010].326000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].326000 msOracleSpatialLayerGetItems was called.
[Tue Mar 16 17:10:45 2010].326000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].326000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].326000 msOracleSpatialLayerClose was called.
Layer: 01E07A90, Layer name: TAAKA_TIENOSA. Layer connection:
user/password at instance
[Tue Mar 16 17:10:45 2010].326000 msOracleSpatialLayerClose. Cleaning
layerinfo handlers.
[Tue Mar 16 17:10:45 2010].326000 msOracleSpatialLayerClose. Release the
Oracle Pool.
[Tue Mar 16 17:10:45 2010].326000
msConnPoolRelease(TAAKA_TIENOSA,user/password at instance,0E12EFE0)
[Tue Mar 16 17:10:45 2010].326000
msConnPoolClose(user/password at instance,0E12EFE0)
[Tue Mar 16 17:10:45 2010].326000 msOracleSpatialLayerOpen called with:
GEOLOC FROM TAAKA.TIENOSA USING UNIQUE TIENOSAID SRID 2393 (Layer
pointer 01E07A90)
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerOpen. Shared
connection not available. Creating one.
[Tue Mar 16 17:10:45 2010].357000
msConnPoolRegister(TAAKA_TIENOSA,user/password at instance,0E12EFE0)
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerGetItems was called.
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerWhichShapes was
called.
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerWhichShapes. Using
this Sql to retrieve the data: SELECT TIENOSAID, TIENOSAID, TIEID,
SILTAID, TIENOSAN_NUMERO, HALTIJA, KUNTA, TUOTTOTAPA, AKTIIVISUUS,
KOHDEKUVAUS, PITUUS, TYOLAJI, PUUSTON_TOIMENPIDE, PINTAMAAN_POISTO,
KALLIO, POHJAN_VAHVISTUS, RUNGON_TIIVISTYS, KANTAVUUS, TURPEEN_PAKSUUS,
VASEN_OJAN_KOKO, OIKEA_OJAN_KOKO, KM_PITUUS, KM_PAKSUUS, KM_LEVEYS,
KM_MAAKERROS, PG_PITUUS, PG_PAKSUUS, PG_LEVEYS, PG_PENGERRYS,
LEIKKAUS_PITUUS, LEIKKAUS_PAKSUUS, LEIKKAUS_LEVEYS, LEIKKAUS_LEIKKAUS,
HUOMAUTUSTEKSTI, MUOKKAAJA, MUOKKAUSPVM, HAKKUU_LEVEYS,
HAKKUU_KOHDEKUVAUS, GEOLOC FROM TAAKA.TIENOSA WHERE SDO_FILTER( GEOLOC,
MDSYS.SDO_GEOMETRY(2003, :srid,
NULL,MDSYS.SDO_ELEM_INFO_ARRAY(1,1003,3),:ordinates
),'querytype=window') = 'TRUE'
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerWhichShapes. Bind
values: srid:2393 minx:3378911.233476 miny:7029122.276619
maxx:3381287.881062 maxy:7031323.687517
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerWhichShapes
getting ordinate definition.
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerWhichShapes
converting to OCIColl.
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerWhichShapes bind
by name and object.
[Tue Mar 16 17:10:45 2010].357000 msOracleSpatialLayerWhichShapes name
and object now bound.
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerNextShape on layer
01E07A90, Fetched 0 more rows (2 total)
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerOpen called with:
GEOLOC FROM TAAKA.TIENOSA USING UNIQUE TIENOSAID SRID 2393 (Layer
pointer 01E07A90)
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerGetItems was called.
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerWhichShapes was
called.
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerWhichShapes():
OracleSpatial error. Cannot allocate layerinfo->items buffer
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerClose was called.
Layer: 01E07A90, Layer name: TAAKA_TIENOSA. Layer connection:
user/password at instance
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerClose. Cleaning
layerinfo handlers.
[Tue Mar 16 17:10:45 2010].372000 msOracleSpatialLayerClose. Release the
Oracle Pool.
[Tue Mar 16 17:10:45 2010].372000
msConnPoolRelease(TAAKA_TIENOSA,user/password at instance,0E12EFE0)
[Tue Mar 16 17:10:45 2010].372000
msConnPoolClose(user/password at instance,0E12EFE0)
[Tue Mar 16 17:10:45 2010].388000 msWFSGetFeature(): WFS server error.
FLTApplyFilterToLayer() failed
[Tue Mar 16 17:10:45 2010].388000 mapserv request processing time
(msLoadMap not incl.): 0.125s
[Tue Mar 16 17:10:45 2010].388000 msFreeMap(): freeing map at 005B6FE8.
[Tue Mar 16 17:10:45 2010].388000 msOracleSpatialLayerClose was called.
Layer: 01E07A90, Layer name: TAAKA_TIENOSA. Layer connection:
user/password at instance
[Tue Mar 16 17:10:45 2010].404000 CGI Request 22 on process 7992
[Tue Mar 16 17:10:45 2010].404000 msOracleSpatialLayerOpen called with:
GEOLOC FROM TAAKA.TIENOSA USING UNIQUE TIENOSAID SRID 2393 (Layer
pointer 0DF74670)
[Tue Mar 16 17:10:45 2010].435000 msOracleSpatialLayerOpen. Shared
connection not available. Creating one.
[Tue Mar 16 17:10:45 2010].435000
msConnPoolRegister(TAAKA_TIENOSA,user/password at instance,01E3A348)
[Tue Mar 16 17:10:45 2010].450000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].450000 msOracleSpatialLayerGetItems was called.
[Tue Mar 16 17:10:45 2010].450000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].450000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].450000 msOracleSpatialLayerClose was called.
Layer: 0DF74670, Layer name: TAAKA_TIENOSA. Layer connection:
user/password at instance
[Tue Mar 16 17:10:45 2010].450000 msOracleSpatialLayerClose. Cleaning
layerinfo handlers.
[Tue Mar 16 17:10:45 2010].450000 msOracleSpatialLayerClose. Release the
Oracle Pool.
[Tue Mar 16 17:10:45 2010].450000
msConnPoolRelease(TAAKA_TIENOSA,user/password at instance,01E3A348)
[Tue Mar 16 17:10:45 2010].450000
msConnPoolClose(user/password at instance,01E3A348)
[Tue Mar 16 17:10:45 2010].450000 msOracleSpatialLayerOpen called with:
GEOLOC FROM TAAKA.TIENOSA USING UNIQUE TIENOSAID SRID 2393 (Layer
pointer 0DF74670)
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerOpen. Shared
connection not available. Creating one.
[Tue Mar 16 17:10:45 2010].482000
msConnPoolRegister(TAAKA_TIENOSA,user/password at instance,01E3A348)
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerGetItems was called.
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerWhichShapes was
called.
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerWhichShapes():
OracleSpatial error. Cannot allocate layerinfo->items buffer
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerClose was called.
Layer: 0DF74670, Layer name: TAAKA_TIENOSA. Layer connection:
user/password at instance
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerClose. Cleaning
layerinfo handlers.
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerClose. Release the
Oracle Pool.
[Tue Mar 16 17:10:45 2010].482000
msConnPoolRelease(TAAKA_TIENOSA,user/password at instance,01E3A348)
[Tue Mar 16 17:10:45 2010].482000
msConnPoolClose(user/password at instance,01E3A348)
[Tue Mar 16 17:10:45 2010].482000 msOracleSpatialLayerOpen called with:
GEOLOC FROM TAAKA.TIENOSA USING UNIQUE TIENOSAID SRID 2393 (Layer
pointer 0DF74670)
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerOpen. Shared
connection not available. Creating one.
[Tue Mar 16 17:10:45 2010].528000
msConnPoolRegister(TAAKA_TIENOSA,user/password at instance,01E3A348)
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerGetItems was called.
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerInitItemInfo was
called.
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerWhichShapes was
called.
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerWhichShapes():
OracleSpatial error. Cannot allocate layerinfo->items buffer
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerFreeItemInfo was
called.
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerClose was called.
Layer: 0DF74670, Layer name: TAAKA_TIENOSA. Layer connection:
user/password at instance
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerClose. Cleaning
layerinfo handlers.
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerClose. Release the
Oracle Pool.
[Tue Mar 16 17:10:45 2010].528000
msConnPoolRelease(TAAKA_TIENOSA,user/password at instance,01E3A348)
[Tue Mar 16 17:10:45 2010].528000
msConnPoolClose(user/password at instance,01E3A348)
[Tue Mar 16 17:10:45 2010].528000 msWFSGetFeature(): WFS server error.
FLTApplyFilterToLayer() failed
[Tue Mar 16 17:10:45 2010].528000 mapserv request processing time
(msLoadMap not incl.): 0.124s
[Tue Mar 16 17:10:45 2010].528000 msFreeMap(): freeing map at 005B6FE8.
[Tue Mar 16 17:10:45 2010].528000 msOracleSpatialLayerClose was called.
Layer: 0DF74670, Layer name: TAAKA_TIENOSA. Layer connection:
user/password at instance
========================
Also I faced that [PROCESSING "CLOSE_CONNECTION=DEFER"] directive in
native Oracle Spatial layer in MapServer as WFS service does not work
and hangs the request.
Vitali.
On 12.03.2010 15:02, Yewondwossen Assefa wrote:
>
> Hi There,
>
> I believe this has been addressed in
> http://trac.osgeo.org/mapserver/ticket/3305. The patches are available
> in the 5.6 branch in svn and should be available for 5.6.2 release. If
> possible can you try building against the svn version. Quick test I
> did with a filter that uses intersect and bbox return expected results:
> ....<Filter><Or><Intersect><PropertyName>Geometry</PropertyName><gml:Polygon><gml:outerBoundaryIs><gml:LinearRing><gml:coordinates>124222,479955
> 132851,479955 132851,483688
> 124222,483688</gml:coordinates></gml:LinearRing></gml:outerBoundaryIs></gml:Polygon></Intersect><BBOX><PropertyName>geometry</PropertyName><Envelope
> srsName='EPSG:28992'><lowerCorner>247268
> 602563</lowerCorner><upperCorner>252296
> 608479</upperCorner></Envelope></BBOX></Or></Filter>....
>
> regards,
>
> Vitali Diatchkov wrote:
>> Hello.
>>
>> I have several issues about MapServer as a WFS service.
>>
>> I am using MapServer as a WFS service from UDIG.
>> Recently I have found that raw <ogc:Intersects> filter doesn't work
>> on latest MapServer that I compile myself against layer with native
>> Oracle Spatial connection. The reason was absence of GEOS support, so
>> I complied MapServer with GEOS. And things became even more unexpected.
>>
>> First of all after switching debug tracing in MAP file I have found
>> that the log file contains:
>> -------------------------------------------------------------------------------
>>
>> [Thu Mar 11 22:03:29 2010].859000
>> msOracleSpatialLayerResultGetShape(): OracleSpatial error.
>> msOracleSpatialLayerResultGetShape record out of range
>> [Thu Mar 11 22:03:29 2010].859000
>> msOracleSpatialLayerResultGetShape(): OracleSpatial error.
>> msOracleSpatialLayerResultGetShape record out of range
>> -------------------------------------------------------------------------------
>>
>>
>> The next step was debugging from Visual Studio and what a surprise.
>> Well it seems native Oracle Spatial connector can directly deal only
>> with WINDOW-based queries (or BBOX). All spatial filters like
>> <ogc:Intersects> are processed in GEOS. But this is implemented in
>> context of legacy architecture and I wouldn't be the first to mention
>> that is is not efficient at all.
>> What I have found that if Intersects filter provided (as an example
>> of non BBOX filter) then during first iteration all records are
>> extracted from the table in msOracleSpatialLayerWhichShapes
>> function. At the same time an extent of the whole layer is used to
>> get those records. If there are 1000000 records, means all are
>> loaded. Intersection is done then using GEOS spatial predicates with
>> a geometry provided in filter for all loaded records. Resuts are
>> somehow cached.
>>
>> Then the next step starts in FLTLayerApplyPlainFilterToLayer calling
>> FLTAddToLayerResultCache. And there a bug is in: a) architecture b)
>> "maporaclespatial.c" logic.
>> Let's assume there are 1300 records in Oracle table. The buffer in
>> msOracleSpatialStatement structure is 1024 by default. When last
>> element in buffer is processed then the next bunch of items are
>> fetched from a result set. In our case first time buffer is fully
>> loaded and has 1024 items. In next fetching the rest 276 items are
>> loaded into the buffer starting from 1st cell. Fetching I am talking
>> about takes place in msOracleSpatialLayerNextShape function.
>> What information is cached?
>> -------------------------------------------------------------
>> addResult(lp->resultcache, shape.classindex, shape.index,
>> shape.tileindex);
>> -------------------------------------------------------------
>> in msQueryByOperator function.
>>
>> In my test case shape.index is a primary key of a feature from a
>> database, like "214789" and shape.tileindex is a kind of index of the
>> cell in a buffer "msOracleSpatialStatemen.obj". Let's assume 5
>> features are actually in a result of intersction operation. In
>> current logic their shape.tileindex value might be 45, 567 or 1238
>> e.g. that is out of mentioned buffer limits. Actually is is assigned in
>> -------------------------------------------------------------------
>> shape->tileindex = sthand->row_num; /* Index into cursor */
>> -------------------------------------------------------------------
>> where row_num just grows permanently.
>>
>> FLTAddToLayerResultCache leads to a call of
>> ----------------------------------------------------------------------
>> int msOracleSpatialLayerResultGetShape( layerObj *layer, shapeObj
>> *shape, int record, long pkey )
>> ----------------------------------------------------------------------
>> where arguments are sort of:
>> record=-1 (always)
>> pkey=214789
>>
>> where "record" is equal to -1 (always, comes from "status =
>> msLayerResultsGetShape(psLayer, &shape, -1, anValues[i]);")
>>
>> this -1 actually kills all the logic in
>> msOracleSpatialLayerResultGetShape.
>>
>> In a result we have nothing to return as a WFS response and dummy XML
>> is output:
>> --------------------------------------------------------------------
>> <wfs:FeatureCollection xsi:schemaLocation="http://www.opengis.net/wfs
>> http://schemas.opengis.net/wfs/1.0.0/WFS-basic.xsd" >
>> <gml:boundedBy>
>> <gml:Box srsName="EPSG:2393">
>> <gml:coordinates>-1.000000,-1.000000
>> -1.000000,-1.000000</gml:coordinates>
>> </gml:Box>
>> </gml:boundedBy>
>> </wfs:FeatureCollection>
>> ------------------------------------------------------------------------------
>>
>>
>> After some studying of code in maporaclespatial.c I can not
>> understand how this scenario might work at all. Is it so that WFS
>> with GEOS and MapServer's native Oracle Spatial is not a solution for
>> this use case and only simple BBOX queries works?
>> But even if to imagine that Intersects filter works and there is no
>> problem with buffer and ietms indexing overlapping, loading of all
>> records during first step for processing by GEOS is not appropriate
>> also.
>>
>> The codebase of MapServer is large and I could miss something
>> important.. Any comments on that?
>>
>> Regards,
>> Vitali Diatchkov.
>>
>> P.S. Latest branch-5-6 codebase is used.
>> _______________________________________________
>> mapserver-dev mailing list
>> mapserver-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>
>
>
More information about the mapserver-dev
mailing list