[QGIS-Developer] Profiler?

[Sebastian M. Ernst]

Hi Nyall, Richard, Nathan,

> Yeah! Full credit goes to Nathan Woodrow here, this panel just exposes
> a profiling tool he added a while back.

I did not know that. Amazing tool.

>> I do have an issue with long startup times on Windows at a client
>> working with a cytrix setup (though I guess it is a general
>> python/startup issue there),
> 
> This is EXACTLY the kind of issue this was added for! I also see
> occasional 5 minute+ startup times on Windows (bare metal installs)

@Richard How does your Citrix setup / architecture look like
(approximately)?

@Nyall If this has anything to do with Python stuff - and it probably
might - can you elaborate on the kind of bare metal setups where you are
observing this?

I am troubleshooting "pure" (i.e. unrelated to QGIS) Python setups for
clients on their Windows boxes on a regular basis. There are multiply
typical culprits for slow load/import times. My #1 favorite is a certain
"security software" typically found in large deployments, but it is a
problem that applies to most "security tools" with varying degrees of
slowdowns. #2 on the list is a mix of NTFS-related issues (rights
management, shadow copies). #1 can usually be identified by
(temporarily) deactivating online scanning of every accessed file. In
some cases, you can only rule it out by deactivating the entire
"security" tool *and* re-installing Python (QGIS in your case) - because
the "security" stuff digs itself deep into the import mechanism. In some
cases you can actually observe changed hash sums of critical Python
libraries ... #2 can be ruled out by installing the entire stack with
*minimal* user rights only (really, really important) onto a clean NTFS
partition / disk without shadow copies. You might not be able to run a
setup.exe with user rights depending on the system configuration -
you'll then have to work around by copying an existing installation into
a user's directory (followed by manually adjusting configurations,
paths, rights and owners). #3 on the list is (SMB/CIFS) network shares.
If your (Python) software is running from there, your best (bad) option
is playing with the buffer configuration of this mechanism, which
usually is not an option in large deployments. Your second best (bad)
option is deactivating the scanning of network traffic in the "security"
tool.

> On that note, 12 seconds is a long time for the plugin load I see in
> your screenshot. Can you expand that out and see if it narrows down
> the culprit?

I'd add that I would also like to understand why loading just the Python
support itself requires 1.5 seconds.

@Nathan & @Nyall Where in QGIS' source can I find the profiling tool? I
just searched and could not locate it ... Would you consider adding
support for a Python profiler to it, e.g. [1], [2] or [3]? The relevant
code could (conditionally) be injected somewhere around here [4] or here
[5]. I have played with this idea a while ago. It works in principle,
but a proper integration into your GUI would be great ...

Best regards,
Sebastian


[1] https://docs.python.org/3/library/profile.html
[2] https://github.com/vpelletier/pprofile
[3] https://github.com/pyutils/line_profiler
[4]
https://github.com/qgis/QGIS/blob/42ba3d78febd5ae9833edc55eac060148c1e0522/src/python/qgspythonutilsimpl.cpp#L208
[5]
https://github.com/qgis/QGIS/blob/42ba3d78febd5ae9833edc55eac060148c1e0522/src/python/qgspythonutilsimpl.cpp#L51