<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi all<br>
<br>
This is not a server specific problem. It is that QGIS may need a
long time to get metadata for some layers. So the best solution is
(as it has been proposed already in this thread) to enhance QGIS
to store the metadata (in the project file) and not read it from
the db the first time. This will also speed up loading of large
projects, it is not only usefull for the server. We should avoid
to make any server specific things in core for that.<br>
<br>
>- there was a bug in QGIS Server which did not use cache
whenever a GetMap request was called with an empty STYLE parameter<br>
<br>
No, the opposite. It did not use cache with a non-empty style
parameter.<br>
<br>
Regards,<br>
Marco<br>
<br>
On 27.02.2016 10:32, kimaidou wrote:<br>
</div>
<blockquote
cite="mid:CAMKXKO4-S6UO5EaOFNULzVEoWDEpUNOrQAY=7T6aqf=z7CwbXQ@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>Hi all,<br>
<br>
</div>
I have made some comparison on requests done when QGIS Server
is called for a simple GetMap, on the first initialization and
aftewards when the cache is found.<br>
<br>
Here are the logs<br>
<br>
When QGIS builds the maplayer object, ie when there is no
cache for this layer yet (first call, for example with a
getcapabilities). This is only done if no cache is found for
this layer in QGIS Server<br>
<br>
</div>
1/ Log when estimated metadata is used (and stats are available
in PostGIS)<br>
<div> <a moz-do-not-send="true"
href="https://paste.sh/yguqyRWL#EPcHHVH3uuijGgEtzH0pAlHb"
target="_blank">https://paste.sh/yguqyRWL#EPcHHVH3uuijGgEtzH0pAlHb</a><br>
<br>
2/ Log when QGIS does not use the metadata for a layer . Which
means this is done for any view at present, since no estimated
metadata is available, or also for a table if the user forgot
to check the option "use metadata" when he first created the
Postgis connection (which can be often the case because this
option is not checked by default in PostgreSQL connection
dialog )<br>
<a moz-do-not-send="true"
href="https://paste.sh/frCglZ9A#A1nQ6BXtdBeEiZWQmTJP8v9l"
target="_blank">https://paste.sh/frCglZ9A#A1nQ6BXtdBeEiZWQmTJP8v9l</a><br>
<br>
3/ Log when a layer is already in QGIS Server cache, for
example for a GetMap request. This means for table and views,
since it does not depend on metadata existence or not, but
only on cache<br>
<a moz-do-not-send="true"
href="https://paste.sh/Rkvqf757#K1MCRitovIxMblBVils1VF7r"
target="_blank">https://paste.sh/Rkvqf757#K1MCRitovIxMblBVils1VF7r</a><br>
<br>
My conclusions here :<br>
<br>
In the first example 1/, we see all the queries QGIS does to
get all the needed information for building the maplayer
object<br>
* postgis information : postgis_version() ,
postgis_geos_version() <br>
* search for topology features -> not really usefull server
side I think ( ? ) <br>
* feature count . In 1, estimated metadata is used, so this
query is fast<br>
</div>
<div> * layer extent calculation . In 1/ QGIS uses
st_estimatedextent, so it also should be fast<br>
<br>
<br>
In the second example 2/, without available stats (no use of
estimated metadata for views or when the option is not checked
in layer ), we can see the performance killing queries (for
complex or big datasets) done by QGIS to get the needed
information :<br>
<br>
* check if the field used as primary key is unique : <br>
SELECT count(distinct ""nb"")=count(""nb"") FROM (SELECT
row_number() OVER() AS nb, * FROM ""2013"".v_geo_parcelle) AS
""subQuery_0"""<br>
<br>
</div>
<div>* Get the extent :<br>
</div>
<div>SELECT st_extent(""geom"") FROM (SELECT row_number() OVER()
AS nb, * FROM ""2013"".v_geo_parcelle) AS ""subQuery_0""<br>
<br>
<br>
</div>
<div>In 3/, QGIS has already built the cache for the maplayer,
and only fetches the features, which is very fast.<br>
</div>
<div><br>
<br>
My thoughts on how to improve QGIS server performance with
PostgreSQL provider (and possibly for other providers as well
)<br>
<br>
</div>
<div>* check that layer cache in QGIS server is well used. <br>
- I do not know for example if the 100 default number is for
the whole server (which is then reached very fast and will
become useless if there are many projects served by QGIS), or
for each projects.<br>
</div>
<div>- there was a bug in QGIS Server which did not use cache
whenever a GetMap request was called with an empty STYLE
parameter. It has been fixed but check your version is not
older to this commit : <a moz-do-not-send="true"
href="https://github.com/qgis/QGIS/commit/a5450a78ffcbba9a75afa8be43460ad741fea050">https://github.com/qgis/QGIS/commit/a5450a78ffcbba9a75afa8be43460ad741fea050</a>
<br>
</div>
<div><br>
* Add an option in QGIS ( in OWS server properties ) to tell
QGIS Server to not get some information if the user who
publish the project knows what he does, meaning<br>
</div>
<div> - use the extent stored in <maplayer> and do not
ask for it server side<br>
</div>
<div> - do NOT check that the primary key is unique -> the
person who built the project and publish it does know that his
layer has a unique primary key.<br>
</div>
<div>Note that there is a limit in the number of layers QGIS
Server will cache ( 100 by default).<br>
</div>
<div><br>
</div>
<div>* Add a even more hard way of telling QGIS Server "Never
delete the cache for this layer, as the extent and the other
properties wont change (even if I allow editing inside my
known layer extent )" . This means the cache will be store
permanently for this layer and would be lost only if the
apache / nginx server is reloaded . I do not know if it is
feasible<br>
<br>
</div>
<div>* Store layers metadata "the QGIS way" in the database, as
we do for styles. This means the users building the project
has enough rights on the postgreSQL database, but could be a
handy QGIS solution for production ready projects. This way we
could store the metadata for tables AND views, in a format
close to what QGIS needs to build the maplayer object. The
user would define some options, such as cache duration in
seconds ( for example, overwrite the cache if the stored
information has more than 1 week, or keep it as long as I do
not manually remove it, via a -1 value for cache duration ).<br>
</div>
<div><br>
</div>
<div>As a conclusion, I think we really need a "QGIS Server
context true/false" when using some of the core classes, such
as maplayer, to be able to do some performance improvements.<br>
<br>
</div>
<div>Régis, René-Luc and myself can have a look at it, if you
need some work done toward this goal.<br>
</div>
<div><br>
</div>
<div>Cheers,<br>
</div>
<div>Michaël<br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2016-02-23 21:52 GMT+01:00 Andreas
Neumann <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:a.neumann@carto.net" target="_blank">a.neumann@carto.net</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">Hi
Alessandro,<span class=""><br>
<br>
On 23.02.2016 17:22, Alessandro Pasotti wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Yes, but Andreas probably meant that plugin support is
still compiled into the server.<br>
</blockquote>
<br>
</span>
yes - if you compile yourself, you can disable it in the
CMAKE settings.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Still, we never measured any noticeable performance
issue with plugins enabled (as long as thre aren't any
of course, otherwise it will obviously depend on what
the plugins do).<br>
</blockquote>
<br>
</span>
yes - only master versions between 2.12 and 2.14 were
affected. It has been fixed meanwhile.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
There was indeed a performance issue with the new access
control implementation (now hopefullt fixed) but IIRC
it's still only in master, to be release soon.<br>
<br>
Also, we aren't aware of any security issue, if you
(Andreas) know about any security issue I would
definitely like to know more about that.<br>
</blockquote>
</span>
I am not aware of any security issues around QGIS server
Python plugins. It is just good to disable stuff if you
don't need it and opt-in if you need it.<span class="HOEnZb"><font
color="#888888"><br>
<br>
Andreas</font></span>
<div class="HOEnZb">
<div class="h5"><br>
_______________________________________________<br>
Qgis-developer mailing list<br>
<a moz-do-not-send="true"
href="mailto:Qgis-developer@lists.osgeo.org"
target="_blank">Qgis-developer@lists.osgeo.org</a><br>
List info: <a moz-do-not-send="true"
href="http://lists.osgeo.org/mailman/listinfo/qgis-developer"
rel="noreferrer" target="_blank">http://lists.osgeo.org/mailman/listinfo/qgis-developer</a><br>
Unsubscribe: <a moz-do-not-send="true"
href="http://lists.osgeo.org/mailman/listinfo/qgis-developer"
rel="noreferrer" target="_blank">http://lists.osgeo.org/mailman/listinfo/qgis-developer</a></div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Qgis-developer mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Qgis-developer@lists.osgeo.org">Qgis-developer@lists.osgeo.org</a>
List info: <a class="moz-txt-link-freetext" href="http://lists.osgeo.org/mailman/listinfo/qgis-developer">http://lists.osgeo.org/mailman/listinfo/qgis-developer</a>
Unsubscribe: <a class="moz-txt-link-freetext" href="http://lists.osgeo.org/mailman/listinfo/qgis-developer">http://lists.osgeo.org/mailman/listinfo/qgis-developer</a></pre>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Dr. Marco Hugentobler
Sourcepole - Linux & Open Source Solutions
Weberstrasse 5, CH-8004 Zürich, Switzerland
<a class="moz-txt-link-abbreviated" href="mailto:marco.hugentobler@sourcepole.ch">marco.hugentobler@sourcepole.ch</a> <a class="moz-txt-link-freetext" href="http://www.sourcepole.ch">http://www.sourcepole.ch</a>
Technical Advisor QGIS Project Steering Committee </pre>
</body>
</html>