[Qgis-developer] Poor asynchronous performance of Python threads?

[Charlie Sharpsteen]

Thanks for the reply Martin!

On Fri, Feb 18, 2011 at 2:07 PM, Martin Dobias <wonder.sk at gmail.com> wrote:
>
> Hi Charlie
>
> On Wed, Feb 9, 2011 at 6:27 AM, Charlie Sharpsteen <chuck at sharpsteen.net> wrote:
> > Hello list,
> > I have been playing around with the development branch of IPython (0.11-dev)
> > as they have built a PyQt-based console that supports all sorts of awesome
> > features like tab-completion, paged browsing of function code and
> > docstrings, and Pygments-powered syntax highlighting in addition to the
> > normal IPython goodness.  I slapped together a quick plugin that adds an
> > IPython-based console to QGIS.  The code is available at:
> >     https://github.com/Sharpie/qgis-ipython
> > To use it, you will need to install ZeroMQ, pyzmq, pygments and the dev
> > version of IPython from:
> >     https://github.com/ipython/ipython
>
> Generally the best / easiest way how to make your code available for
> testing for others is to upload the plugin at pyqgis.org to the
> contributed repository. Requiring development version of ipython would
> be however a show stopper for most users...


I would love to see a finished plugin eventually released to PyQGIS.
However, the code is currently little more than a proof-of-concept
prototype and is also based on a development version of IPython, so I
didn't feel like a public release on PyQGIS was appropriate at this
time.


> > The console fires up allright, but unfortunately very, very, very slow.
> >  There is about ~1-2 seconds of lag between invoking a command
> > and receiving a response.  For comparison, I threw together a minimal PyQGIS
> > application that also uses the IPython console:
> >     https://gist.github.com/817914
> > The PyQGIS version is very snappy and there is no lag.  I suspect the
> > difference is due to the PyQt widget is talking to an external IPython
> > process through ZeroMQ.  When a command is executed, the console sends a
> > message and then a Python thread waits for the response.  With PyQGIS,
> > Python is controlling the execution of the entire application.  However,
> > when the console is used as a plugin it is running in an embedded
> > interpreter and it looks like QGIS takes a while to schedule the execution
> > of the Python threads.
> > Does anyone have any suggestions for solving or working around this issue?
>
> An interesting issue. Why do you think that IPython uses another
> process?


The interesting thing about IPython is that in addition to being a
really nice Python shell it is also a framework for parallel and
distributed computing.  The 0.11 series splits the interpreter into a
separate process using ZMQ so that a single console could dispatch
jobs to several independent interpreters and those interpreters could
be running on entirely different machines.

When this framework matures, it could open up some really interesting
possibilities for dispatching computationally intensive operations
from QGIS.


> Running the console in another process would disable the
> communication between QGIS interface and the console - and that is not
> really wanted.


At the moment, my main interest in getting IPython into QGIS is so
that it can help me develop Python plugins.  The existing Python
console is very nice, but I keep tapping the tab key in vain whenever
I forget what attributes an object has.  Currently I am working on an
IPython kernel that does not run in a separate process so it can
access objects like `qgis.utils.iface`.  Progress is slow, but
promising.

However, it would also be nice to leave the option of spawning
out-of-process shells for computationally intensive jobs.


>Maybe you could try to run that code in a profiler to
> find out where is the bottleneck.
>
> Martin


I have given this a couple of tries.  Unfortunately I don't have much
experience using profilers so I wasn't able to learn much.  I think
the question I need to answer is: Is the delivery of messages being
delayed by the QGIS C++ code, the Python interpreter or the ZMQ event
loop?  At this point it *seems* too have something to do with the C++
code as everything runs fine in a pure PyQGIS app.

Some tools I know about that I have access to are gprof, dtrace and
valgrind---is there one that you would suggest as being particularly
suited to this task?  Do I need to recompile QGIS with some special
flags?


-Charlie