WFS performance issue

[Steve Lime]

You also need to know how long it takes outside of WFS context, that is,
how long
a normal MapServer query would run...

Steve

>>> Bart van den Eijnden <bartvde at XS4ALL.NL> 3/30/2005 5:30:20 AM >>>
Hi Assefa,

I did a WFS GetFeature BBOX test on a 3.2 million record table in
PostGIS.

Mapserver 4.4.1: 170 seconds
Mapserver CVS: varies between 60 and 85 seconds

Still seems like an improvable performance to me, but where is the
bottleneck in Mapserver WFS?

WMS GetMap on the same bbox takes half a second.

I have attached the PostgreSQL log file.

Best regards,
Bart

> Arnulf,
>
>   I have added the initial support to improve the performance (see
> details in bug 1292 :
> http://mapserver.gis.umn.edu/bugs/show_bug.cgi?id=1292).
>
>   If you have a chance to do some testing using Postgis or Oracle
please
> let me know if it adresses some of the performance issues.
>
> Later,
>
> Arnulf Christl wrote:
>> Frank Warmerdam wrote:
>> [...]
>>
>>> My understanding is that there are two issues.  One is the
unnecessary
>>> or
>>> confusing spatial query imposed by the .map bbox even when the wfs
>>> query has no spatial restriction.  The other is the fact that
>>> attribute filtering
>>> is done as a post-processing step by mapserver instead of being
passed
>>> into the RDBMS as a WHERE style filter.
>>>
>>> I propose to dig into making MapServer queries able to handle NULL
>>> query rects in cases where no spatial filtering makes sense.  This
can
>>> reduce query optimization confusion in some spatial databases (and
>>> OGR!)
>>> as well as help avoid problems like Tom K. ran into recently with
the
>>> unexpected effect of his .map EXTENT on WFS queries.
>>>
>>> Assefa is offering to rewrite WFS attribute queries in a backend
>>> specific
>>> format (likely just adding support for SQL WHERE for Oracle,
PostGIS
>>> and OGR) so that the attribute filtering can go down to the
database
>>> instead of being done in MapServer as a postprocessing step.
>>>
>>> So I believe it is Assefa's change that will make a several orders
>>> of magnitude improvement in performance for your case.  My change
>>> might also be helpful in some cases.
>>>
>>>
>>>> The fun thing is that we *can* do it by using MapServer as a
warped
>>>> WMS.
>>>> We simply activate the internal (not OGC) FILTER expression which
we
>>>> add
>>>> to the WMS getMap as a vendor parameter. Setting DUMP=true you get
a
>>>> valid GML - which is all we need to gazetteer, highlight, download
and
>>>> edit the land parcel.
>>>
>>>
>>>
>>> Wow, thats pretty neat though I don't follow it completely.  But
>>> hopefully
>>> we will reach a point where things can be a little easier to
optimize
>>> "out of the box".
>>>
>>> Best regards,
>>
>>
>> To get an impression of what we are talking about you can have a
look
>> here, it went online yesterday.
>> http://www.flo.rlp.de/
>> Its all in German though, ...
>>
>> Start with <Demo Zugang> and enter <demo> as user and pwd. The
showcase
>> boasts a reduced dataset, contents are faked, the aerial photography
res
>> is a meager 1m/pixel. It is a closed system for around 20T farmers
in
>> Rhineland Palatinate. Within the closed system aerials have a res
of
>> 25cm/pixel and the database hosts the full set of 6.5 million land
>> parcels.
>>
>> Besides rendering the aerials (Mr.Sid), land parcels (Oracle), etc.
as
>> WMS we use MapServer as WFS for finding land parcels. From the left
>> registers select <Flurstuecke> (land parcels) and enter a 4 into
the
>> field <Stueck> (as it is a demo thats about all you can do). The
request
>> returns the land parcel as GML, Mapbender transforms it into
JavaScript
>> onmouseover and click events, so that clicking on it centers the map
and
>> onmouseover highlights the geometry. The highlighting is done by
sending
>> the transformed GML to a server side PHP which renders the land
parcel
>> with a simple red outline and returns the image as overlay. So
maybe
>> this might be a little slow and impractical across the atlantic. I
am
>> very interested in performance comments from your side of the
pond...
>>
>> We do this for all searches. It is fantastic how easy things get
when
>> you just stick to using standards (we just had a discussion on why
is
>> Open Source using/pushing Open Standards). Dump any geometry into
the
>> WFS (or slightly clobbered MapServer) and go search for it. The
little
>> details take the time, if you search for the place name <Koblenz>
>> (register Ort) it works if you search for <koblenz> it wont find
>> anything. Tribute to a rather dumb shape/dbf behind as data.
>>
>> Inside we go one step further, the farmer can digitize new objects
and
>> store them in a central PostgreSQL/PostGIS, currently directly as
WKT as
>> MapServer doesn't want the T inside WFS. We talked about this one
and we
>> will probably stick with GeoServer WFS-T - so Barts suggestion might
be
>> a good idea.
>>
>> In the lower left corner you find a button named <Schlag> (field).
It
>> returns all fields farmed by the current user as GML - to be
downloaded
>> and edited/whatever with udig, jump... This is a simple MapServer
WFS
>> outlet.
>>
>> The productive system goes through https and is wrapped by a J2EE
>> webstart application that manages user authentication (using an
Open
>> LDAP) and authorization by attributing a role in Mapbender. Besides
it
>> load balances the system. The customer got us to duplicate
everything(!)
>> on one GNU/Linux (quad Xeon) and two Windows servers (double
Xeons).
>> Imagine all this Mr.Sid and Oracle connectivity crap and compiling
not
>> on one operating system only but on two different ones... Anyway it
>> works fine so that we can attest MapServer to work excellent with
both
>> OS (if we had a QM section on the MapServer homepage this could be
>> published there).
>>
>> We are really grateful for you guys (all of us) doing all this good
>> work. We just fumble and tie the pieces together and the outcome
>> invariably is a great and stable application. Sorry for my undamped
>> enthusiasm, but I cannot stress this fact often enough!
>>
>> Best Arnulf.
>>
>> --
>> ------------------------
>> Arnulf B. Christl
>> ------------------------
>> http://www.ccgis.org
>> http://www.mapbender.org
>> ------------------------
>>
>
> --
> ----------------------------------------------------------------
> Assefa Yewondwossen
> Software Analyst
>
> Email: assefa at dmsolutions.ca
> http://www.dmsolutions.ca/
>
> Phone: (613) 565-5056 (ext 14)
> Fax:   (613) 565-0925
> ----------------------------------------------------------------
>