WFS performance issue

Wed Mar 30 07:32:14 EST 2005

Strange things are happening for my testcase (unfinished GML etc.) so
discard my test results until I find out what's going on exactly.

Best regards,
Bart

> Hi Assefa,
>
> I did a WFS GetFeature BBOX test on a 3.2 million record table in PostGIS.
>
> Mapserver 4.4.1: 170 seconds
> Mapserver CVS: varies between 60 and 85 seconds
>
> Still seems like an improvable performance to me, but where is the
> bottleneck in Mapserver WFS?
>
> WMS GetMap on the same bbox takes half a second.
>
> I have attached the PostgreSQL log file.
>
> Best regards,
> Bart
>
>> Arnulf,
>>
>>   I have added the initial support to improve the performance (see
>> details in bug 1292 :
>> http://mapserver.gis.umn.edu/bugs/show_bug.cgi?id=1292).
>>
>>   If you have a chance to do some testing using Postgis or Oracle please
>> let me know if it adresses some of the performance issues.
>>
>> Later,
>>
>> Arnulf Christl wrote:
>>> Frank Warmerdam wrote:
>>> [...]
>>>
>>>> My understanding is that there are two issues.  One is the unnecessary
>>>> or
>>>> confusing spatial query imposed by the .map bbox even when the wfs
>>>> query has no spatial restriction.  The other is the fact that
>>>> attribute filtering
>>>> is done as a post-processing step by mapserver instead of being passed
>>>> into the RDBMS as a WHERE style filter.
>>>>
>>>> I propose to dig into making MapServer queries able to handle NULL
>>>> query rects in cases where no spatial filtering makes sense.  This can
>>>> reduce query optimization confusion in some spatial databases (and
>>>> OGR!)
>>>> as well as help avoid problems like Tom K. ran into recently with the
>>>> unexpected effect of his .map EXTENT on WFS queries.
>>>>
>>>> Assefa is offering to rewrite WFS attribute queries in a backend
>>>> specific
>>>> format (likely just adding support for SQL WHERE for Oracle, PostGIS
>>>> and OGR) so that the attribute filtering can go down to the database
>>>> instead of being done in MapServer as a postprocessing step.
>>>>
>>>> So I believe it is Assefa's change that will make a several orders
>>>> of magnitude improvement in performance for your case.  My change
>>>> might also be helpful in some cases.
>>>>
>>>>
>>>>> The fun thing is that we *can* do it by using MapServer as a warped
>>>>> WMS.
>>>>> We simply activate the internal (not OGC) FILTER expression which we
>>>>> add
>>>>> to the WMS getMap as a vendor parameter. Setting DUMP=true you get a
>>>>> valid GML - which is all we need to gazetteer, highlight, download
>>>>> and
>>>>> edit the land parcel.
>>>>
>>>>
>>>>
>>>> Wow, thats pretty neat though I don't follow it completely.  But
>>>> hopefully
>>>> we will reach a point where things can be a little easier to optimize
>>>> "out of the box".
>>>>
>>>> Best regards,
>>>
>>>
>>> To get an impression of what we are talking about you can have a look
>>> here, it went online yesterday.
>>> http://www.flo.rlp.de/
>>> Its all in German though, ...
>>>
>>> Start with <Demo Zugang> and enter <demo> as user and pwd. The showcase
>>> boasts a reduced dataset, contents are faked, the aerial photography
>>> res
>>> is a meager 1m/pixel. It is a closed system for around 20T farmers in
>>> Rhineland Palatinate. Within the closed system aerials have a res of
>>> 25cm/pixel and the database hosts the full set of 6.5 million land
>>> parcels.
>>>
>>> Besides rendering the aerials (Mr.Sid), land parcels (Oracle), etc. as
>>> WMS we use MapServer as WFS for finding land parcels. From the left
>>> registers select <Flurstuecke> (land parcels) and enter a 4 into the
>>> field <Stueck> (as it is a demo thats about all you can do). The
>>> request
>>> returns the land parcel as GML, Mapbender transforms it into JavaScript
>>> onmouseover and click events, so that clicking on it centers the map
>>> and
>>> onmouseover highlights the geometry. The highlighting is done by
>>> sending
>>> the transformed GML to a server side PHP which renders the land parcel
>>> with a simple red outline and returns the image as overlay. So maybe
>>> this might be a little slow and impractical across the atlantic. I am
>>> very interested in performance comments from your side of the pond...
>>>
>>> We do this for all searches. It is fantastic how easy things get when
>>> you just stick to using standards (we just had a discussion on why is
>>> Open Source using/pushing Open Standards). Dump any geometry into the
>>> WFS (or slightly clobbered MapServer) and go search for it. The little
>>> details take the time, if you search for the place name <Koblenz>
>>> (register Ort) it works if you search for <koblenz> it wont find
>>> anything. Tribute to a rather dumb shape/dbf behind as data.
>>>
>>> Inside we go one step further, the farmer can digitize new objects and
>>> store them in a central PostgreSQL/PostGIS, currently directly as WKT
>>> as
>>> MapServer doesn't want the T inside WFS. We talked about this one and
>>> we
>>> will probably stick with GeoServer WFS-T - so Barts suggestion might be
>>> a good idea.
>>>
>>> In the lower left corner you find a button named <Schlag> (field). It
>>> returns all fields farmed by the current user as GML - to be downloaded
>>> and edited/whatever with udig, jump... This is a simple MapServer WFS
>>> outlet.
>>>
>>> The productive system goes through https and is wrapped by a J2EE
>>> webstart application that manages user authentication (using an Open
>>> LDAP) and authorization by attributing a role in Mapbender. Besides it
>>> load balances the system. The customer got us to duplicate
>>> everything(!)
>>> on one GNU/Linux (quad Xeon) and two Windows servers (double Xeons).
>>> Imagine all this Mr.Sid and Oracle connectivity crap and compiling not
>>> on one operating system only but on two different ones... Anyway it
>>> works fine so that we can attest MapServer to work excellent with both
>>> OS (if we had a QM section on the MapServer homepage this could be
>>> published there).
>>>
>>> We are really grateful for you guys (all of us) doing all this good
>>> work. We just fumble and tie the pieces together and the outcome
>>> invariably is a great and stable application. Sorry for my undamped
>>> enthusiasm, but I cannot stress this fact often enough!
>>>
>>> Best Arnulf.
>>>
>>> --
>>> ------------------------
>>> Arnulf B. Christl
>>> ------------------------
>>> http://www.ccgis.org
>>> http://www.mapbender.org
>>> ------------------------
>>>
>>
>> --
>> ----------------------------------------------------------------
>> Assefa Yewondwossen
>> Software Analyst
>>
>> Email: assefa at dmsolutions.ca
>> http://www.dmsolutions.ca/
>>
>> Phone: (613) 565-5056 (ext 14)
>> Fax:   (613) 565-0925
>> ----------------------------------------------------------------
>>
>