[QGIS-Developer] Monitor QGIS (Server) memory consumption per project

[kimaidou]

Hey,

I updated my "audit qgis projects" processing algorithm with a
quick-and-dirty method to calculate the memory usage, before and after each
project reading
https://gist.github.com/mdouchin/cd89a259d3560635d4bb8708ad430caa#file-lizmapauditqgisprojects-py-L150

It is not very reliable and should not be used in a highly changing
environment (production server with high activity).
It could help nonetheless...

Regards,
Michaël

Le mer. 28 juil. 2021 à 12:08, kimaidou <kimaidou at gmail.com> a écrit :

> Hi all,
>
> We (@3liz.com) use QGIS Server since a long time to provide hosting
> services for our clients, with Lizmap Web Client (LWC) as the web map
> builder.
>
> Having hundred LWC instances, we pay obviously attention to metrics, and
> monitor the CPU, memory, requests response time, etc.
>
> We sometimes encounter heavy memory consumption for specific projects, for
> example :
> https://github.com/qgis/QGIS/issues/43787
> which put a lot of strain to the servers.
>
> It is often very hard to find the project(s) responsible for a high memory
> usage, in a production context, when many requests are run in parallel for
> many projects.
>
> I created a POC processing script to audit all the QGIS projects stored in
> a folder:
> https://gist.github.com/mdouchin/cd89a259d3560635d4bb8708ad430caa/
> which helps a bit to gather some key figures, such as the number of
> layers, the sum of the images used in map layouts, etc.
> This can help, but often the memory consumption is not related to the
> number of layers, and we use a "find a needle in the haystack" approach to
> find the problematic project(s).
>
> For example, I spent once a whole day finding that a too big JPEG used in
> a print layout causes the heavy memory usage:
> * look at the memory curve,
> * find the moment when the memory increased a lot
> * use the command line find  to get which QGIS project files has changed
> during this period (for example because it has been uploaded by the user)
> * look inside QGIS logs to find which project has been loaded during this
> period, and compare
> * if you are lucky, there are not so many projects involved...
> * download the project, open it in QGIS Desktop to observe and confirm the
> memory issue
> * try to understand which causes the issue
> * ...
>
> We are also building a REST API over py-qgis-server (
> https://github.com/3liz/py-qgis-server ) which will allow to get
> information on the loaded projects, to ease the auditing process in
> real-time. (It will be opensource).
>
> In the processing script, and in this API tool, we still miss a very
> important metric: how much each project consumes the system memory (RAM). I
> am not sure it could be easily done, but it will be very handy to:
>
> * find heavy memory consumer projects
> * see how the memory evolves
> * compare the memory usage for each QGIS versions (with the same project
> and layers)
> * help to find the possible memory leaks
>
> This message is not a feature request, I would like to get your experience
> and opinion regarding this memory usage issue. Any contribution welcome ;-)
>
> Thanks in advance
> Michaël
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210728/d8989f4b/attachment.html>