WFS performance issue

Wed Mar 30 06:30:20 EST 2005

Hi Assefa,

I did a WFS GetFeature BBOX test on a 3.2 million record table in PostGIS.

Mapserver 4.4.1: 170 seconds
Mapserver CVS: varies between 60 and 85 seconds

Still seems like an improvable performance to me, but where is the
bottleneck in Mapserver WFS?

WMS GetMap on the same bbox takes half a second.

I have attached the PostgreSQL log file.

Best regards,
Bart

> Arnulf,
>
>   I have added the initial support to improve the performance (see
> details in bug 1292 :
> http://mapserver.gis.umn.edu/bugs/show_bug.cgi?id=1292).
>
>   If you have a chance to do some testing using Postgis or Oracle please
> let me know if it adresses some of the performance issues.
>
> Later,
>
> Arnulf Christl wrote:
>> Frank Warmerdam wrote:
>> [...]
>>
>>> My understanding is that there are two issues.  One is the unnecessary
>>> or
>>> confusing spatial query imposed by the .map bbox even when the wfs
>>> query has no spatial restriction.  The other is the fact that
>>> attribute filtering
>>> is done as a post-processing step by mapserver instead of being passed
>>> into the RDBMS as a WHERE style filter.
>>>
>>> I propose to dig into making MapServer queries able to handle NULL
>>> query rects in cases where no spatial filtering makes sense.  This can
>>> reduce query optimization confusion in some spatial databases (and
>>> OGR!)
>>> as well as help avoid problems like Tom K. ran into recently with the
>>> unexpected effect of his .map EXTENT on WFS queries.
>>>
>>> Assefa is offering to rewrite WFS attribute queries in a backend
>>> specific
>>> format (likely just adding support for SQL WHERE for Oracle, PostGIS
>>> and OGR) so that the attribute filtering can go down to the database
>>> instead of being done in MapServer as a postprocessing step.
>>>
>>> So I believe it is Assefa's change that will make a several orders
>>> of magnitude improvement in performance for your case.  My change
>>> might also be helpful in some cases.
>>>
>>>
>>>> The fun thing is that we *can* do it by using MapServer as a warped
>>>> WMS.
>>>> We simply activate the internal (not OGC) FILTER expression which we
>>>> add
>>>> to the WMS getMap as a vendor parameter. Setting DUMP=true you get a
>>>> valid GML - which is all we need to gazetteer, highlight, download and
>>>> edit the land parcel.
>>>
>>>
>>>
>>> Wow, thats pretty neat though I don't follow it completely.  But
>>> hopefully
>>> we will reach a point where things can be a little easier to optimize
>>> "out of the box".
>>>
>>> Best regards,
>>
>>
>> To get an impression of what we are talking about you can have a look
>> here, it went online yesterday.
>> http://www.flo.rlp.de/
>> Its all in German though, ...
>>
>> Start with <Demo Zugang> and enter <demo> as user and pwd. The showcase
>> boasts a reduced dataset, contents are faked, the aerial photography res
>> is a meager 1m/pixel. It is a closed system for around 20T farmers in
>> Rhineland Palatinate. Within the closed system aerials have a res of
>> 25cm/pixel and the database hosts the full set of 6.5 million land
>> parcels.
>>
>> Besides rendering the aerials (Mr.Sid), land parcels (Oracle), etc. as
>> WMS we use MapServer as WFS for finding land parcels. From the left
>> registers select <Flurstuecke> (land parcels) and enter a 4 into the
>> field <Stueck> (as it is a demo thats about all you can do). The request
>> returns the land parcel as GML, Mapbender transforms it into JavaScript
>> onmouseover and click events, so that clicking on it centers the map and
>> onmouseover highlights the geometry. The highlighting is done by sending
>> the transformed GML to a server side PHP which renders the land parcel
>> with a simple red outline and returns the image as overlay. So maybe
>> this might be a little slow and impractical across the atlantic. I am
>> very interested in performance comments from your side of the pond...
>>
>> We do this for all searches. It is fantastic how easy things get when
>> you just stick to using standards (we just had a discussion on why is
>> Open Source using/pushing Open Standards). Dump any geometry into the
>> WFS (or slightly clobbered MapServer) and go search for it. The little
>> details take the time, if you search for the place name <Koblenz>
>> (register Ort) it works if you search for <koblenz> it wont find
>> anything. Tribute to a rather dumb shape/dbf behind as data.
>>
>> Inside we go one step further, the farmer can digitize new objects and
>> store them in a central PostgreSQL/PostGIS, currently directly as WKT as
>> MapServer doesn't want the T inside WFS. We talked about this one and we
>> will probably stick with GeoServer WFS-T - so Barts suggestion might be
>> a good idea.
>>
>> In the lower left corner you find a button named <Schlag> (field). It
>> returns all fields farmed by the current user as GML - to be downloaded
>> and edited/whatever with udig, jump... This is a simple MapServer WFS
>> outlet.
>>
>> The productive system goes through https and is wrapped by a J2EE
>> webstart application that manages user authentication (using an Open
>> LDAP) and authorization by attributing a role in Mapbender. Besides it
>> load balances the system. The customer got us to duplicate everything(!)
>> on one GNU/Linux (quad Xeon) and two Windows servers (double Xeons).
>> Imagine all this Mr.Sid and Oracle connectivity crap and compiling not
>> on one operating system only but on two different ones... Anyway it
>> works fine so that we can attest MapServer to work excellent with both
>> OS (if we had a QM section on the MapServer homepage this could be
>> published there).
>>
>> We are really grateful for you guys (all of us) doing all this good
>> work. We just fumble and tie the pieces together and the outcome
>> invariably is a great and stable application. Sorry for my undamped
>> enthusiasm, but I cannot stress this fact often enough!
>>
>> Best Arnulf.
>>
>> --
>> ------------------------
>> Arnulf B. Christl
>> ------------------------
>> http://www.ccgis.org
>> http://www.mapbender.org
>> ------------------------
>>
>
> --
> ----------------------------------------------------------------
> Assefa Yewondwossen
> Software Analyst
>
> Email: assefa at dmsolutions.ca
> http://www.dmsolutions.ca/
>
> Phone: (613) 565-5056 (ext 14)
> Fax:   (613) 565-0925
> ----------------------------------------------------------------
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: WFS GetFeature BBOX.log
Type: application/octet-stream
Size: 8226 bytes
Desc: not available
Url : http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20050330/4288f577/WFSGetFeatureBBOX.obj