[Qgis-developer] QEP: QGIS Mapserver Python Plugins

Tue Sep 9 01:59:06 PDT 2014

2014-09-06 12:50 GMT+02:00 Martin Dobias <wonder.sk at gmail.com>:
> Hi Alessandro
>
> Finally I had some time to look at your proposal (your mail and the
> proposed QEP [1]). I am not that much involved in the QGIS server
> development, but the Python word in the thread subject always draws my
> attention :-)

Hello Martin,

thank you for your comments!

Answers follow.

>
> On Thu, Aug 28, 2014 at 7:49 PM, Alessandro Pasotti <apasotti at gmail.com> wrote:
>>
>> The rationale behind server plugins is doublefold: first they could
>> provide additional services without the need to touch the C++
>> codebase, second they allow for GUI-based configuration since the
>> server plugins are not separated from the desktop plugins (of course
>> the environment and permissions should be carefully configured to
>> allow information sharing from the desktop user to the webserver user
>> ).
>
> I understand the first reason (use python instead of c++ to extend the
> server), but I am not sure I see the second reason (allow GUI-based
> configuration). Actually I find that mixing of desktop and server
> plugin functionality is not making things any better than having two
> separate plugins, one for desktop and one for server. They are meant
> to be running in two completely different environments, it may be a
> good idea to treat them in such way. Still I guess having some kind of
> plugin functionality also in server may be useful.

Configuration of QGIS Server from the Desktop is already implemented:
you can configure QGIS OWS Server from the the desktop today and
everybody finds it useful.
Also, I've been really astonished by some of my customers when I knew
that they are currently running QGIS Desktop on their (virtualized)
server machines in order to manage their QGIS projects and configure
the OWS QGIS server. The conclusion is that I'm not sure anymore about
the religious separation of server and client machines (I would still
keep them separate but many others don't).

But I insist that having the possibility to manage a single plugin for
both the server and the desktop side is an opportunity, a possibility
that will not enforce authors to use it.
If you (as a plugin author) will want to split the server and desktop
side of a plugin, you will be perfectly allowed to do it: you will
just choose a different name for the server and desktop side of your
plugin.

Another good reason to have the same plugins machinery for the server
and the client is that all the services to manage plugins are already
in place: plugins website, plugin installer etc., even configuration
testing from QGIS desktop would be easier to implement when it's all
in one place.

>
> At the same time I think in many cases it would be actually better to
> have the functionality in python plugin (e.g. WPS server based on
> processing framework) completely separated from QGIS server and run it
> through an ordinary interface for Python (WSGI). One could even employ
> some Apache rewriting rules to make the impression that the service is
> still handled by QGIS server.

My proposal is not about any particular service and WPS was just an
example, again, this is about the possibility/opportunity to expand
QGIS server functionalities through plugins written in Python.
It's not me (and probably not you)  who knows how plugin authors will
exploit this opportunity, maybe they will write 1 million LOC
programs, maybe they will write tiny glue scripts to other programs,
who knows?

What I like of the idea of having server plugins managed through the
desktop is that the users will immediately see the available server
plugins and their updates and they will be able to install/upgrade
them with a click and they will be able to configure them with a nice
GUI. This is not something for me, for you or for the majority of the
readers of this mailing list: we like vim and editing rc files from
the shell. This is something for the average end users and believe me:
they are not always system administrators anymore.

>
> You mention that this is just an early prototype to see where things
> can go, so I understand some things are quite hacky / experimental. If
> the team is happy to proceed with the concept of plugins for QGIS
> server, here are some things I think need changing compared to the
> current proposal/implementation:

Absolutely agreed. What we need now is a committing step that gives a
green light to Python server plugins.
With that green light we can start to discuss implementation details.
And of course we will need another green light when the details will
be fully defined.

> - we need proper object-oriented approach for the plugins. Like QGIS
> desktop provides QgisInterface instance for plugins, QGIS server
> should also provide some kind of QgsServerInterface which would be
> used by server plugins. Such interface would give the plugin access to
> all data structures it needs (e.g. parsed project).

Yes, I thought about that but it was beyond the scope of this
proof-of-concept implementation.
I guess that we would need a different iface object for the server, I
definitely need some help in this area, your advice and help is warmly
welcome!

> - in the server there is already QgsOWSServer interface which existing
> services derive from. Plugins should be also base on that interface
> (may need some further work)

I didn't consider using that interface (which is probably virtual
anyway), but I feel that using *only* that interface could be a
limitation, think about plugins that do not provide an OWS service or
think about a simple regexp replace plugin that replaces a string
altering the (XML) response.

> - the output from plugin should be abstracted - in the server there is
> already QgsRequestHandler interface for such things (may need some
> further work)

Agreed, see also my final notes.

> - loading of plugins should be done in a very similar way to desktop
> plugins - similar to "classFactory(iface)" entry point, there could be
> "serverClassFactory(iface)" entry point that returns instance of
> running server plugin

Agreed. This is exactly what I would like to see (you help would be
highly appreciated on this topic).

> - I don't like the fact that plugins declare their "service" and
> "methods" in metadata.txt. I think plugins should register their
> QgsOWSServer implementation to server interface when they are loaded.
> This is more flexible - a plugin may register more services or provide
> some extra info during the registration

Yes, this in a nice idea, even if it's much more complex than just
register the available service and methods, store them in an hash and
call that methods if they match, in that case all the heavy job of url
parsing and method checking is done by C++ routines. The current
implementation is based on speed: all what that server has to do is
match the SERVICE parameter against an hash of registered plugins.
We need to keep in our mind that we are on the server side and
performance is a concern, choosing another system for plugin service
registration is fine as long it doesn't cause unacceptable delays when
the server has to find a plugin that matches the request parameters.
We should also consider that keeping service/methods in metadata could
allow for automatic GetCapabilities requests handling but I don't make
this a point.

>
> Cheers
> Martin
>
> [1] https://github.com/qgis/QGIS-Enhancement-Proposals/pull/3

Some final notes:

I think we should first agree on the fact that implementing Python
plugins for the server is fine, as long as the impact on the current
server performances is negligible or zero in case there are no plugins
enabled.

Second, we should decide if we want tight or loose bindings with QGIS
server request handling: in the former case, I would implement this
within signals/slots that are invoked

1. when application starts (before entering FCGI loop)
2. when the request starts (after GET/POST parsing) the plugin can
manipulate the request and issue a response (think about
authentication/authorization plugins)
3. when the response is complete (and here we would probably need
headers separated from the body), before it is sent to the client
4. at 404 (when the loop has finished without a match for C++ OWS
services), this is not strictly necessary because the previous slot
could also catch this case, but I feel this case is more clear and
moreover it is only invoked when there is no match with existing OWS
C++ core services.

I still don't know wether a signal/slot implementation (or other
similar listener patterns) will have an acceptable impact on
performances if compared with a pure 404 handler which has for sure no
impact, I guess this is not a problem but maybe someone has better
ideas.

Looking forward to read your reply and comments from others.

Regards.

-- 
Alessandro Pasotti
w3:   www.itopen.it