[Qgis-developer] [Server] performance questions

Sun Feb 28 23:40:08 PST 2016

Hi all

This is not a server specific problem. It is that QGIS may need a long 
time to get metadata for some layers. So the best solution is (as it has 
been proposed already in this thread) to enhance QGIS to store the 
metadata (in the project file) and not read it from the db the first 
time. This will also speed up loading of large projects, it is not only 
usefull for the server. We should avoid to make any server specific 
things in core for that.

 >- there was a bug in QGIS Server which did not use cache whenever a 
GetMap request was called with an empty STYLE parameter

No, the opposite. It did not use cache with a non-empty style parameter.

Regards,
Marco

On 27.02.2016 10:32, kimaidou wrote:
> Hi all,
>
> I have made some comparison on requests done when QGIS Server is 
> called for a simple GetMap, on the first initialization and aftewards 
> when the cache is found.
>
> Here are the logs
>
> When QGIS builds the maplayer object, ie when there is no cache for 
> this layer yet (first call, for example with a getcapabilities). This 
> is only done if no cache is found for this layer in QGIS Server
>
> 1/ Log when estimated metadata is used (and stats are available in 
> PostGIS)
> https://paste.sh/yguqyRWL#EPcHHVH3uuijGgEtzH0pAlHb
>
> 2/ Log when QGIS does not use the metadata for a layer . Which means 
> this is done for any view at present, since no estimated metadata is 
> available, or also for a table if the user forgot to check the option 
> "use metadata" when he first created the Postgis connection (which can 
> be often the case because this option is not checked by default in 
> PostgreSQL connection dialog )
> https://paste.sh/frCglZ9A#A1nQ6BXtdBeEiZWQmTJP8v9l
>
> 3/ Log when a layer is already in QGIS Server cache, for example for a 
> GetMap request. This means for table and views, since it does not 
> depend on metadata existence or not, but only on cache
> https://paste.sh/Rkvqf757#K1MCRitovIxMblBVils1VF7r
>
> My conclusions here :
>
> In the first example 1/, we see all the queries QGIS does to get all 
> the needed information for building the maplayer object
> * postgis information : postgis_version() , postgis_geos_version()
> * search for topology features -> not really usefull server side I 
> think ( ? )
> * feature count . In 1, estimated metadata is used, so this query is fast
> * layer extent calculation . In 1/ QGIS uses st_estimatedextent, so it 
> also should be fast
>
>
> In the second example 2/, without available stats (no use of estimated 
> metadata for views or when the option is not checked in layer ), we 
> can see the performance killing queries (for complex or big datasets) 
> done by QGIS to get the needed information :
>
> * check if the field used as primary key is unique :
> SELECT count(distinct ""nb"")=count(""nb"") FROM (SELECT row_number() 
> OVER() AS nb, * FROM ""2013"".v_geo_parcelle) AS ""subQuery_0"""
>
> * Get the extent :
> SELECT st_extent(""geom"") FROM (SELECT row_number() OVER() AS nb, * 
> FROM ""2013"".v_geo_parcelle) AS ""subQuery_0""
>
>
> In 3/, QGIS has already built the cache for the maplayer, and only 
> fetches the features, which is very fast.
>
>
> My thoughts on how to improve QGIS server performance with PostgreSQL 
> provider (and possibly for other providers as well )
>
> * check that layer cache in QGIS server is well used.
> - I do not know for example if the 100 default number is for the whole 
> server (which is then reached very fast and will become useless if 
> there are many projects served by QGIS), or for each projects.
> - there was a bug in QGIS Server which did not use cache whenever a 
> GetMap request was called with an empty STYLE parameter. It has been 
> fixed but check your version is not older to this commit : 
> https://github.com/qgis/QGIS/commit/a5450a78ffcbba9a75afa8be43460ad741fea050 
>
>
> * Add an option in QGIS ( in OWS server properties ) to tell QGIS 
> Server to not get some information if the user who publish the project 
> knows what he does, meaning
>   - use the extent stored in <maplayer> and do not ask for it server side
>   - do NOT check that the primary key is unique -> the person who 
> built the project and publish it does know that his layer has a unique 
> primary key.
> Note that there is a limit in the number of layers QGIS Server will 
> cache ( 100 by default).
>
> * Add a even more hard way of telling QGIS Server "Never delete the 
> cache for this layer, as the extent and the other properties wont 
> change (even if I allow editing inside my known layer extent )" . This 
> means the cache will be store permanently for this layer and would be 
> lost only if the apache / nginx server is reloaded . I do not know if 
> it is feasible
>
> * Store layers metadata "the QGIS way" in the database, as we do for 
> styles. This means the users building the project has enough rights on 
> the postgreSQL database, but could be a handy QGIS solution for 
> production ready projects. This way we could store the metadata for 
> tables AND views, in a format close to what QGIS needs to build the 
> maplayer object. The user would define some options, such as cache 
> duration in seconds ( for example, overwrite the cache if the stored 
> information has more than 1 week, or keep it as long as I do not 
> manually remove it, via a -1 value for cache duration ).
>
> As a conclusion, I think we really need a "QGIS Server context 
> true/false" when using some of the core classes, such as maplayer, to 
> be able to do some performance improvements.
>
> Régis, René-Luc and myself can have a look at it, if you need some 
> work done toward this goal.
>
> Cheers,
> Michaël
>
> 2016-02-23 21:52 GMT+01:00 Andreas Neumann <a.neumann at carto.net 
> <mailto:a.neumann at carto.net>>:
>
>     Hi Alessandro,
>
>     On 23.02.2016 17:22, Alessandro Pasotti wrote:
>
>
>
>         Yes, but Andreas probably meant that plugin support is still
>         compiled into the server.
>
>
>     yes - if you compile yourself, you can disable it in the CMAKE
>     settings.
>
>
>         Still, we never measured any noticeable performance issue with
>         plugins enabled (as long as thre aren't any of course,
>         otherwise it will obviously depend on what the plugins do).
>
>
>     yes - only master versions between 2.12 and 2.14 were affected. It
>     has been fixed meanwhile.
>
>
>         There was indeed a performance issue with the new access
>         control implementation (now hopefullt fixed) but IIRC it's
>         still only in master, to be release soon.
>
>         Also, we aren't aware of any security issue, if you (Andreas)
>         know about any security issue I would definitely like to know
>         more about that.
>
>     I am not aware of any security issues around QGIS server Python
>     plugins. It is just good to disable stuff if you don't need it and
>     opt-in if you need it.
>
>     Andreas
>
>     _______________________________________________
>     Qgis-developer mailing list
>     Qgis-developer at lists.osgeo.org <mailto:Qgis-developer at lists.osgeo.org>
>     List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>     Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
>
>
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

-- 
Dr. Marco Hugentobler
Sourcepole -  Linux & Open Source Solutions
Weberstrasse 5, CH-8004 Zürich, Switzerland
marco.hugentobler at sourcepole.ch http://www.sourcepole.ch
Technical Advisor QGIS Project Steering Committee

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20160229/6bbf035e/attachment-0001.html>