[Qgis-developer] Changes to python/utils.py

[Larry Shaffer]

Hi Martin,

Thanks for getting back to me about this.

On Sat, Jun 23, 2012 at 2:36 PM, Martin Dobias <wonder.sk at gmail.com> wrote:
> Hi Larry
>
> and sorry for getting back to you so late.
>
> On Wed, May 30, 2012 at 6:07 PM, Larry Shaffer <larrys at dakotacarto.com> wrote:
>> A little background... I have spent some effort trying to get my
>> plugins to fully unload/load as many times as possible within QGIS
>> (via Plugin Reloader) before having to restart the app. Inevitably any
>> complex plugins crash (usually QGIS, too) because something is not
>> being removed from sys.modules during the unload of the plugin, using
>> the current utils.py module tracking method.
>
> Hmm... are you aware of any reproducible crash? I would be interested
> to see where it comes from. The unloader and import hook with
> dependency tracking is not bullet-proof, but it should cover the
> common cases quite well. You have probably seen that there is also
> explicit logic for unloading resource data from pyrcc4-generated files
> - without that reloaded plugins tend to crash due to dangling pointer
> to resources - and that cannot be handled by a generic reloader.

I have not done much testing to isolate the crashes. Mostly I have
been consistently reloading my Plugin Editor plugin which has complex
imports and many modules. It tends to crash after ~4-5 reloads,
specifically when trying open the plugin's GUI after it has been
reloaded. Broken pointers to previously not-unloaded resources or
objects looks like the culprit.

I'm thinking to debug this, several specifically built test plugins
with different import and resource setups, and an auto-reloading
plugin that runs on a timer and inspects/logs aspects of the test
plugins' import dependency graphs during reloads would be at good
starting point. It's a little too complicated to debug without some
type of test harness.

Don't have time now, but will look into this after I release my
plugin, because I think it will be a source of instability for some
users when trying to use my Plugin Editor for development.


>> I now see your point on hard vs. live reloading of plugins (delete
>> from sys.modules vs. reload). Yes, avoid live reloading of plugins =
>> good thing, keep sanity. :^)  Still, I think there is something to
>> gain from reloader.py's more extensive dependency tracking
>> methodology.
>>
>> In utils.py, the import wrapper takes a conservative approach, by only
>> including package-dependent plugin modules. I think reloader.py's
>> approach, with its global parent, creates a more complete dependency
>> graph. In my preliminary tests, all modules were reloaded (including
>> those with relative imports or sys.path augmentation for embedded 3rd
>> party packages).
>>
>> If reloader.py's dependency graphing approach was used for utils.py's
>> import wrapper, for cleaner plugin unloads, then at least two things
>> would have to happen beyond what utils.py does now:
>>
>> 1) On deletion from sys.modules ( _unloadPluginModules() ), important
>> modules would need to be skipped (qgis, PyQt4, utils, console, etc.).
>> I had to do this with the reloader.py implementation I am currently
>> using, which is for live reloading when testing normal
>> modules/packages, not plugins.
>
> There could be a module that does all the mandatory imports, so that
> would create another branch in the dependency graph, not allowing
> those modules to be unloaded, right?

Yes, that might work. Also, I think there are few enough of the known
core-don't-reload modules to just skip them on a reload by comparison
of whether their package name is in a list, which is what has worked
with my use of reloader.py.

My patched reloader.py [0] chains with qgis.utils, instead of
replacing its __import__ function. Then, reloader is turned on for the
import of modules for testing/running/reloading inside the plugin.
Allows for editing and re-running code directly from within Plugin
Editor, without affecting the normal means of unloading/loading other
plugins.

[0] http://dl.dropbox.com/u/4058089/qgis/editor_reloader.py.diff


>> 2) A cross-reference check of some kind to make sure the module to be
>> removed is not already in another plugin's dependency graph. The
>> developer could then be notified that a clean reload was not possible,
>> and why. Since this is a plugin-testing reload, a developer is
>> probably ok with turning off other plugins. Normal plugin unloads
>> (Plugin Manager, Installer, etc.) could silently ignore the incomplete
>> unload, as they do now.
>
> Or eventually it could try to reload the other plugins that are
> dependent of the shared python modules.

Since I see reloading as really just a step in the development process
of a plugin, and not a regular user type of operation, I don't see an
issue with reloading other plugins with same dependencies. Maybe an
optional method to do this? Then a developer can choose to reload
other plugins or not.


Regards,

Larry Shaffer
Dakota Cartography
Black Hills, South Dakota