[UMN_MAPSERVER-USERS] Question re: PostGIS and MapScript getExtent() function

Steve Hall steve.hall at MERCATORGEOSYSTEMS.CO.UK
Tue Jul 12 18:11:12 EDT 2005


It sounds like for Oracle adding the "Fast" option is pretty easy - read the 
metadata table and return that (with the caveat described below that it's 
only as good as the metadata used to create the layer in Oracle in the first 

As Fernando points out this can be wrong - it is not validated by Oracle in 
anyway.   The Oracle "shp2sdo" tool however will populate the table 
correctly so any loads based on Shape files should (to begin with anyway!) 
have correct values.

I guess the easiest way to do this is to add "Fast" functions for all data 
sources, which, to begin with at least, could simply call the existing 
implementations (i.e the new ms*LayerGetExtentFast() simply calls the 
current ms*LayerGetExtent().   For Oracle and PostGIS though the Fast and 
"non-Fast" functions would have significantly different implementations re: 
what we've discussed.  I know nothing about the WMS code but I would guess 
that could be changed pretty easily to call the new *Fast functions without 
fear of breakage, as to begin with all they would do is call the existing 

Would the *Fast functions need to be exposed to MapScript in anyway or would 
they really only be useful in a WMS context?


----- Original Message ----- 
From: "Fernando Simon" <fsimon at univali.br>
To: "Steve Hall" <steve.hall at MERCATORGEOSYSTEMS.CO.UK>
Sent: Tuesday, July 12, 2005 10:42 PM
Subject: Re: [UMN_MAPSERVER-USERS] Question re: PostGIS and MapScript 
getExtent() function

> Hi folks,
>   I added the support for getExtent function in 4.4 version and can run 
> using two ways: using MBR and CONVEXHULL database function. The bugs: 1136 
> and 1112 is related about this issue.
>   The column DIMINFO can store the extent for the table, but it's don't 
> guarantee that it is the real extent, in many cases it's store: 
> +180, -180, -90 and +90. So, when an user apply a filter or any 
> restriction this values can be very wrong. Of course that I can use this 
> for a fast extent, but it's not guarantee that it's real, many users can 
> report in the future that the extents it is not real....
>   The MBR function can be very slow when the table store many data, I 
> needed to use it for 9i version and geodetic values. Convexhull function 
> is more fast, but cannot be apply for 9i version. I used these functions 
> because an user can change a layer filter and call the getextent function 
> without need to change the data parameter for the layer. The performance 
> is very good for the most requests, and increase with a good spatial 
> index.
>   The idea about the fast extent is very nice, more hints for this issue? 
> Only for WMS, like Paul suggest? What is the impact for this change in the 
> others Mapserv modules? Hints?
>   Thanks again.
> ------------------------------------------------------------------------
> Fernando Simon
> Mapserver and Oracle Spatial developer
> G10 - Laboratorio de Computacao Aplicada - Brazil
> http://g10.cttmar.univali.br - UNIVALI/CTTMAR
> ------------------------------------------------------------------------
> Steve Hall wrote:
>> In my experience with Oracle it too can be pretty slow getting an MBR on 
>> a table that big  ;-)
>> I don't know much about other data sources, but Oracle's spatial metadata 
>> registry (USER_SDO_GEOM_METADATA) does maintain layer extent data in the 
>> DIMINFO column so could be retrieved without needing to perform a spatial 
>> aggregate, and hence would be a suitable candidate for the *Fast function 
>> you descibe here.
>> Like you say PostGIS also has a couple of choices in this area.
>> I guess by definition the *Fast functions would just ignore, (or fail?) 
>> if a filter was specified...?   Or perhaps if a filter is specified, the 
>> *Fast functions could call the non-Fast functions to get the answer in 
>> the (blind?) hope that the filter makes them sufficiently performant.
>> Thanks,
>> Steve
>> ----- Original Message ----- From: "Paul Ramsey" 
>> <pramsey at refractions.net>
>> To: "Steve Hall" <steve.hall at mercatorgeosystems.co.uk>
>> Sent: Tuesday, July 12, 2005 7:37 PM
>> Subject: Re: [UMN_MAPSERVER-USERS] Question re: PostGIS and MapScript 
>> getExtent() function
>>> Steve,
>>> Try it on a 20M record table in Oracle. Both implementations,  Oracle's 
>>> and your uncommented PostGIS one, do the same functional  thing, which 
>>> is a full table scan.  It is possible that Oracle is  slightly smarter 
>>> about reading the index for the AGGR_MBR case, which  might make it work 
>>> better on large tables: one for the testing range.
>>> I think your tuppence might be right on the money:
>>> - Enable ms*GetLayerExtent() to provide the "right" answer in all 
>>> cases, even if it is expensive.
>>> - Add a ms*GetLayerExtentFast() that for the easy data sources just 
>>> calls GetLayerExtent, and for the hard data sources does whatever 
>>> combination of fast hacks and cop-outs are required.
>>> WMS GetCapabilities should call GetLayerExtentFast and mapscript  should 
>>> call GetLayerExtent.
>>> Unrelated question for the gurus: does GetLayerExtent respect filters 
>>> for all data sources?
>>> Paul
>>> On 12-Jul-05, at 11:30 AM, Steve Hall wrote:
>>>> Thanks Paul,
>>>> This all sounds very reasonable to me....  I'll keep this to myself 
>>>> for now then.
>>>> I'm curious about the fact that this is implemented for Oracle  which 
>>>> is performing a very similar operation (i.e a spatial  aggregate, 
>>>> SDO_AGGR_MBR).  Is Oracle more efficient than PostGIS in  this area or 
>>>> is there a difference in approach that i'm missing...?
>>>> For my two-pennorth a function like this is very useful, as it  allows 
>>>> me to change the filter for a layer, and alter the map  extent to match 
>>>> the extent of the "new" layer, without needing to  do any "dirty" SQL 
>>>> on the back-end database manually to find the  new extent.  Perhaps if 
>>>> it can't / shouldn't be done via these  msXXXLayerGetExtent() functions 
>>>> there is another way, or could it  be listed as an enhancement?
>>>> Thanks again, i'm new to the contributing side of Open Source so 
>>>> forgive my early mistakes!
>>>> Steve
>>>> ----- Original Message ----- From: "Paul Ramsey" 
>>>> <pramsey at refractions.net>
>>>> To: "Steve Hall" <steve.hall at mercatorgeosystems.co.uk>
>>>> Sent: Tuesday, July 12, 2005 6:09 PM
>>>> Subject: Re: [UMN_MAPSERVER-USERS] Question re: PostGIS and  MapScript 
>>>> getExtent() function
>>>>> Steve,
>>>>> If you grep through the source, you'll see the calls, which are 
>>>>> mainly in the WMS code (pulling extents for capabilities).  The 
>>>>> problem is that the return time for the call can be very large  for 
>>>>> very large tables.  Which is not really acceptable for a  capabilities 
>>>>> return. Maybe it is acceptable for your  application.  This is the 
>>>>> real problem.  We want something with  very low return times (as a 
>>>>> shapefile extents call is) but we  cannot get it with the extent() 
>>>>> function, because that requires a  sequence scan.
>>>>> So basically, we cannot simply stick in your implementation, or  we 
>>>>> are building a big obscure "gotcha" into mapserver. ("Why does  it 
>>>>> take forever to return a capabilities file for my PostGIS  backed WMS? 
>>>>> Everything else is fast.")
>>>>> Perhaps the folks on -DEV can return some zen wisdom on the  correct 
>>>>> approach to this.
>>>>> Paul
>>>>> On 12-Jul-05, at 9:52 AM, Steve Hall wrote:
>>>>>> Hi Paul,
>>>>>> Exactly right - it definitely needs reviewing by those that know 
>>>>>> more about MapServer internals than I do.  (which is most of you  I 
>>>>>> expect).
>>>>>> How is this function used my MapServer itself then?  The context  in 
>>>>>> which I am using it is via an explicit $layerObj->getExtent()  call 
>>>>>> from PHP/MapScript, so in this respect the current  implementation 
>>>>>> is not much use, but I suspect there is more to  this function than 
>>>>>> just an implementation for this call.   The  main issue I had was 
>>>>>> that this function worked as I expected for  Oracle, yet not for 
>>>>>> PostGIS, meaning I'd need some ugly code in  my app to deal with the 
>>>>>> inconsistency...
>>>>>> Anyway, i've attached my version of the function for you to take  a 
>>>>>> look at. My implemenation is basically the version that was 
>>>>>> commented out, with the compilcation errors & warning resolved, 
>>>>>> added support for the FILTER expression and support for either  BOX 
>>>>>> or BOX3D return clauses from PostGIS extent().
>>>>>> Many thanks,
>>>>>> Steve
>>>>>> ----- Original Message ----- From: "Paul Ramsey" 
>>>>>> <pramsey at refractions.net>
>>>>>> To: "Steve Hall" <steve.hall at MERCATORGEOSYSTEMS.CO.UK>
>>>>>> Sent: Tuesday, July 12, 2005 3:49 PM
>>>>>> Subject: Re: [UMN_MAPSERVER-USERS] Question re: PostGIS and 
>>>>>> MapScript getExtent() function
>>>>>>> msPOSTGISLayerGetExtent() is potentially extremely inefficient, 
>>>>>>> depending on how it is implemented, so I would have to see your 
>>>>>>> patch to judge if it does more harm than good. It may be doing 
>>>>>>> good things for you and bad things for others.  One thing it 
>>>>>>> *could* do without harm is read the wms_extent meta-parameter  and 
>>>>>>> use that if it is provided by the user. There are also a  few 
>>>>>>> version specific fast extent functions available in  PostGIS, but 
>>>>>>> they are not guaranteed to work in all cases, at  all times, 
>>>>>>> correctly.
>>>>>>> On 12-Jul-05, at 2:58 AM, Steve Hall wrote:
>>>>>>>> Hi All,
>>>>>>>> I'm using MapServer 4.6.0 with Oracle Spatial and PostGIS data 
>>>>>>>> sources via PHP/MapScript.  Part of my application code looks 
>>>>>>>> like this pseudo-code :
>>>>>>>>     $map->setLayerFilter("ThematicData", "REQUEST_ID=121");
>>>>>>>>     $map->zoomToLayer("ThematicData");
>>>>>>>> Here i'm setting a specific FILTER expression for a layer, and 
>>>>>>>> then zooming to the extent of the (now filtered) layer.  I use 
>>>>>>>> the layerObj->getExtent() and then mapObj->setExtent()  functions 
>>>>>>>> to achieve this behind the scenes.  This works great  for Oracle, 
>>>>>>>> the query is fast and makes for nice, tidy code in  my  application 
>>>>>>>> without the need for Oracle specific SQL  statements  to derive the 
>>>>>>>> new extent.
>>>>>>>> However when I try this on PostGIS I just get a massive   (useless) 
>>>>>>>> extent returned.  Digging into the code I can see  that  the 
>>>>>>>> msPOSTGISLayerGetExtent() function in mappostgis.c  is  effectively 
>>>>>>>> doing nothing although a lot of code is  commented out.
>>>>>>>> Working on my local code I have re-enabled this code, fixed the 
>>>>>>>> compilation errors and changed it so that when a FILTER is 
>>>>>>>> applied to the layer this is used as part of the SQL call to   PG. 
>>>>>>>> It seems to work great (from PHP/MapScript) and  performance  is 
>>>>>>>> fine and is really useful for me meaning I  don't need to 
>>>>>>>> calculate the extents for a map myself -  MapServer can supply  the 
>>>>>>>> information in a generic and  abstract fashion.
>>>>>>>> Ideally I would like to get this code fed back into MapServer, 
>>>>>>>> but of course only if it's acceptable to you all.  So,   should  I 
>>>>>>>> instead post this to mapserver-dev or can I pass on  the new  code 
>>>>>>>> to someone from here?
>>>>>>>> There may be good, historical, reasons why this wasn't ever 
>>>>>>>> implemented though so please let me know!
>>>>>>>> Many thanks,
>>>>>>>> Steve Hall
>>>>>>>> <msPOSTGISLayerGetExtent.c>

More information about the mapserver-dev mailing list