Question re: PostGIS and MapScript getExtent() function
Steve Hall
steve.hall at MERCATORGEOSYSTEMS.CO.UK
Tue Jul 12 15:11:12 PDT 2005
Hi,
It sounds like for Oracle adding the "Fast" option is pretty easy - read the
metadata table and return that (with the caveat described below that it's
only as good as the metadata used to create the layer in Oracle in the first
place).
As Fernando points out this can be wrong - it is not validated by Oracle in
anyway. The Oracle "shp2sdo" tool however will populate the table
correctly so any loads based on Shape files should (to begin with anyway!)
have correct values.
I guess the easiest way to do this is to add "Fast" functions for all data
sources, which, to begin with at least, could simply call the existing
implementations (i.e the new ms*LayerGetExtentFast() simply calls the
current ms*LayerGetExtent(). For Oracle and PostGIS though the Fast and
"non-Fast" functions would have significantly different implementations re:
what we've discussed. I know nothing about the WMS code but I would guess
that could be changed pretty easily to call the new *Fast functions without
fear of breakage, as to begin with all they would do is call the existing
implementations....
Would the *Fast functions need to be exposed to MapScript in anyway or would
they really only be useful in a WMS context?
Steve
----- Original Message -----
From: "Fernando Simon" <fsimon at univali.br>
To: "Steve Hall" <steve.hall at MERCATORGEOSYSTEMS.CO.UK>
Cc: <MAPSERVER-USERS at LISTS.UMN.EDU>; <pramsey at REFRACTIONS.NET>
Sent: Tuesday, July 12, 2005 10:42 PM
Subject: Re: [UMN_MAPSERVER-USERS] Question re: PostGIS and MapScript
getExtent() function
> Hi folks,
> I added the support for getExtent function in 4.4 version and can run
> using two ways: using MBR and CONVEXHULL database function. The bugs: 1136
> and 1112 is related about this issue.
> The column DIMINFO can store the extent for the table, but it's don't
> guarantee that it is the real extent, in many cases it's store:
> +180, -180, -90 and +90. So, when an user apply a filter or any
> restriction this values can be very wrong. Of course that I can use this
> for a fast extent, but it's not guarantee that it's real, many users can
> report in the future that the extents it is not real....
> The MBR function can be very slow when the table store many data, I
> needed to use it for 9i version and geodetic values. Convexhull function
> is more fast, but cannot be apply for 9i version. I used these functions
> because an user can change a layer filter and call the getextent function
> without need to change the data parameter for the layer. The performance
> is very good for the most requests, and increase with a good spatial
> index.
> The idea about the fast extent is very nice, more hints for this issue?
> Only for WMS, like Paul suggest? What is the impact for this change in the
> others Mapserv modules? Hints?
> Thanks again.
>
> ------------------------------------------------------------------------
> Fernando Simon
> Mapserver and Oracle Spatial developer
> G10 - Laboratorio de Computacao Aplicada - Brazil
> http://g10.cttmar.univali.br - UNIVALI/CTTMAR
> ------------------------------------------------------------------------
>
>
> Steve Hall wrote:
>
>> In my experience with Oracle it too can be pretty slow getting an MBR on
>> a table that big ;-)
>>
>> I don't know much about other data sources, but Oracle's spatial metadata
>> registry (USER_SDO_GEOM_METADATA) does maintain layer extent data in the
>> DIMINFO column so could be retrieved without needing to perform a spatial
>> aggregate, and hence would be a suitable candidate for the *Fast function
>> you descibe here.
>>
>> Like you say PostGIS also has a couple of choices in this area.
>>
>> I guess by definition the *Fast functions would just ignore, (or fail?)
>> if a filter was specified...? Or perhaps if a filter is specified, the
>> *Fast functions could call the non-Fast functions to get the answer in
>> the (blind?) hope that the filter makes them sufficiently performant.
>>
>> Thanks,
>>
>> Steve
>>
>>
>> ----- Original Message ----- From: "Paul Ramsey"
>> <pramsey at refractions.net>
>> To: "Steve Hall" <steve.hall at mercatorgeosystems.co.uk>
>> Cc: <MAPSERVER-USERS at LISTS.UMN.EDU>; <MAPSERVER-DEV at LISTS.UMN.EDU>
>> Sent: Tuesday, July 12, 2005 7:37 PM
>> Subject: Re: [UMN_MAPSERVER-USERS] Question re: PostGIS and MapScript
>> getExtent() function
>>
>>
>>> Steve,
>>>
>>> Try it on a 20M record table in Oracle. Both implementations, Oracle's
>>> and your uncommented PostGIS one, do the same functional thing, which
>>> is a full table scan. It is possible that Oracle is slightly smarter
>>> about reading the index for the AGGR_MBR case, which might make it work
>>> better on large tables: one for the testing range.
>>>
>>> I think your tuppence might be right on the money:
>>>
>>> - Enable ms*GetLayerExtent() to provide the "right" answer in all
>>> cases, even if it is expensive.
>>> - Add a ms*GetLayerExtentFast() that for the easy data sources just
>>> calls GetLayerExtent, and for the hard data sources does whatever
>>> combination of fast hacks and cop-outs are required.
>>>
>>> WMS GetCapabilities should call GetLayerExtentFast and mapscript should
>>> call GetLayerExtent.
>>>
>>> Unrelated question for the gurus: does GetLayerExtent respect filters
>>> for all data sources?
>>>
>>> Paul
>>>
>>> On 12-Jul-05, at 11:30 AM, Steve Hall wrote:
>>>
>>>> Thanks Paul,
>>>>
>>>> This all sounds very reasonable to me.... I'll keep this to myself
>>>> for now then.
>>>>
>>>> I'm curious about the fact that this is implemented for Oracle which
>>>> is performing a very similar operation (i.e a spatial aggregate,
>>>> SDO_AGGR_MBR). Is Oracle more efficient than PostGIS in this area or
>>>> is there a difference in approach that i'm missing...?
>>>>
>>>> For my two-pennorth a function like this is very useful, as it allows
>>>> me to change the filter for a layer, and alter the map extent to match
>>>> the extent of the "new" layer, without needing to do any "dirty" SQL
>>>> on the back-end database manually to find the new extent. Perhaps if
>>>> it can't / shouldn't be done via these msXXXLayerGetExtent() functions
>>>> there is another way, or could it be listed as an enhancement?
>>>>
>>>> Thanks again, i'm new to the contributing side of Open Source so
>>>> forgive my early mistakes!
>>>>
>>>> Steve
>>>>
>>>>
>>>>
>>>> ----- Original Message ----- From: "Paul Ramsey"
>>>> <pramsey at refractions.net>
>>>> To: "Steve Hall" <steve.hall at mercatorgeosystems.co.uk>
>>>> Cc: <MAPSERVER-USERS at LISTS.UMN.EDU>; <MAPSERVER-DEV at LISTS.UMN.EDU>
>>>> Sent: Tuesday, July 12, 2005 6:09 PM
>>>> Subject: Re: [UMN_MAPSERVER-USERS] Question re: PostGIS and MapScript
>>>> getExtent() function
>>>>
>>>>
>>>>
>>>>> Steve,
>>>>>
>>>>> If you grep through the source, you'll see the calls, which are
>>>>> mainly in the WMS code (pulling extents for capabilities). The
>>>>> problem is that the return time for the call can be very large for
>>>>> very large tables. Which is not really acceptable for a capabilities
>>>>> return. Maybe it is acceptable for your application. This is the
>>>>> real problem. We want something with very low return times (as a
>>>>> shapefile extents call is) but we cannot get it with the extent()
>>>>> function, because that requires a sequence scan.
>>>>>
>>>>> So basically, we cannot simply stick in your implementation, or we
>>>>> are building a big obscure "gotcha" into mapserver. ("Why does it
>>>>> take forever to return a capabilities file for my PostGIS backed WMS?
>>>>> Everything else is fast.")
>>>>>
>>>>> Perhaps the folks on -DEV can return some zen wisdom on the correct
>>>>> approach to this.
>>>>>
>>>>> Paul
>>>>>
>>>>> On 12-Jul-05, at 9:52 AM, Steve Hall wrote:
>>>>>
>>>>>
>>>>>> Hi Paul,
>>>>>>
>>>>>> Exactly right - it definitely needs reviewing by those that know
>>>>>> more about MapServer internals than I do. (which is most of you I
>>>>>> expect).
>>>>>>
>>>>>> How is this function used my MapServer itself then? The context in
>>>>>> which I am using it is via an explicit $layerObj->getExtent() call
>>>>>> from PHP/MapScript, so in this respect the current implementation
>>>>>> is not much use, but I suspect there is more to this function than
>>>>>> just an implementation for this call. The main issue I had was
>>>>>> that this function worked as I expected for Oracle, yet not for
>>>>>> PostGIS, meaning I'd need some ugly code in my app to deal with the
>>>>>> inconsistency...
>>>>>>
>>>>>> Anyway, i've attached my version of the function for you to take a
>>>>>> look at. My implemenation is basically the version that was
>>>>>> commented out, with the compilcation errors & warning resolved,
>>>>>> added support for the FILTER expression and support for either BOX
>>>>>> or BOX3D return clauses from PostGIS extent().
>>>>>>
>>>>>> Many thanks,
>>>>>>
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>>
>>>>>> ----- Original Message ----- From: "Paul Ramsey"
>>>>>> <pramsey at refractions.net>
>>>>>> To: "Steve Hall" <steve.hall at MERCATORGEOSYSTEMS.CO.UK>
>>>>>> Cc: <MAPSERVER-USERS at LISTS.UMN.EDU>
>>>>>> Sent: Tuesday, July 12, 2005 3:49 PM
>>>>>> Subject: Re: [UMN_MAPSERVER-USERS] Question re: PostGIS and
>>>>>> MapScript getExtent() function
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> msPOSTGISLayerGetExtent() is potentially extremely inefficient,
>>>>>>> depending on how it is implemented, so I would have to see your
>>>>>>> patch to judge if it does more harm than good. It may be doing
>>>>>>> good things for you and bad things for others. One thing it
>>>>>>> *could* do without harm is read the wms_extent meta-parameter and
>>>>>>> use that if it is provided by the user. There are also a few
>>>>>>> version specific fast extent functions available in PostGIS, but
>>>>>>> they are not guaranteed to work in all cases, at all times,
>>>>>>> correctly.
>>>>>>>
>>>>>>> On 12-Jul-05, at 2:58 AM, Steve Hall wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> I'm using MapServer 4.6.0 with Oracle Spatial and PostGIS data
>>>>>>>> sources via PHP/MapScript. Part of my application code looks
>>>>>>>> like this pseudo-code :
>>>>>>>>
>>>>>>>> $map->setLayerFilter("ThematicData", "REQUEST_ID=121");
>>>>>>>> $map->zoomToLayer("ThematicData");
>>>>>>>> Here i'm setting a specific FILTER expression for a layer, and
>>>>>>>> then zooming to the extent of the (now filtered) layer. I use
>>>>>>>> the layerObj->getExtent() and then mapObj->setExtent() functions
>>>>>>>> to achieve this behind the scenes. This works great for Oracle,
>>>>>>>> the query is fast and makes for nice, tidy code in my application
>>>>>>>> without the need for Oracle specific SQL statements to derive the
>>>>>>>> new extent.
>>>>>>>>
>>>>>>>> However when I try this on PostGIS I just get a massive (useless)
>>>>>>>> extent returned. Digging into the code I can see that the
>>>>>>>> msPOSTGISLayerGetExtent() function in mappostgis.c is effectively
>>>>>>>> doing nothing although a lot of code is commented out.
>>>>>>>>
>>>>>>>> Working on my local code I have re-enabled this code, fixed the
>>>>>>>> compilation errors and changed it so that when a FILTER is
>>>>>>>> applied to the layer this is used as part of the SQL call to PG.
>>>>>>>> It seems to work great (from PHP/MapScript) and performance is
>>>>>>>> fine and is really useful for me meaning I don't need to
>>>>>>>> calculate the extents for a map myself - MapServer can supply the
>>>>>>>> information in a generic and abstract fashion.
>>>>>>>>
>>>>>>>> Ideally I would like to get this code fed back into MapServer,
>>>>>>>> but of course only if it's acceptable to you all. So, should I
>>>>>>>> instead post this to mapserver-dev or can I pass on the new code
>>>>>>>> to someone from here?
>>>>>>>>
>>>>>>>> There may be good, historical, reasons why this wasn't ever
>>>>>>>> implemented though so please let me know!
>>>>>>>>
>>>>>>>> Many thanks,
>>>>>>>>
>>>>>>>> Steve Hall
>>>>>>>>
>>>>>>>>
>>>>>>>> <msPOSTGISLayerGetExtent.c>
>>>>>>>>
>>>>
>>>>
>>
>
More information about the MapServer-users
mailing list