[Qgis-developer] Multi threading and background processing

Wed Nov 20 15:00:50 PST 2013

Hi Angus

We have met for sure.  How ya doing?  Yes, the simulation model will be
open source :).  In the longer term it would be nice to make this be part
of a larger forest simulation, including links to a spatial management
model, but that is not the immediate priority.

Thanks for the tips about multiple processes and shared memory.  Hmmmm... I
suspect that it might be slightly tidier to keep this all within one
process, at least I would like to give it a try to start with.  Once I have
built up enough callous tissue on my forehead maybe I'll come back to this.

As you can imagine from my newbie questions I'm not a python guy.  While I
really like a lot of the modern mechanisms in python I have never been able
to look at a python script without being puzzled by the retro/hipster
nature of using whitespace as syntax.  As someone who once lined up fortran
statements on punch cards you will have to pry whitespace indifferent
languages from my cold hunt-n-peck fingers. Anyway, I'm expecting to use
python for the ui and glue code, and do all of the number crunching in a C
module.  Horses for courses eh?

Good point about the global interpreter lock.  This might not be a problem
because I should be able to instantiate the simulator in the main
interpreter, and then kick it off into its own compute thread (PyQt QThread
I guess?).  It looks like the GIL gets released when entering compiled code
(
http://stackoverflow.com/questions/1595649/threading-in-a-pyqt-application-use-qt-threads-or-python-threads)<http://stackoverflow.com/questions/1595649/threading-in-a-pyqt-application-use-qt-threads-or-python-threads>,and
I should be able to use the Qt QueuedConnnection signal/slot mechanism from
within C to safely communicate back and forth between the worker thread and
the app.  But regardless, the simulation code wont need to interact with
python once it has been initialized other than a few housekeeping bits that
can probably be handled using signals and handlers in the C code (send
status to app, accept pause/continue/terminate in the worker thread, etc).
I think I'm OK here.

With respect to my previous point 2) I'm hoping that I can subclass the
raster data source, and add a QMutex/critical section around any calls to
fetch data through GDALDataset::RasterIO.  The simulator would use the same
critical section around the places where it is modifying shared data
structures.

I assume that I handle 3) using a signal/slot mechanism: set the
dataChanged() state on the map layer in the simulation thread and then send
a qt signal to the app that it needs to refresh the map canvas.  Can anyone
confirm if this is the way to do it?

As for my previous point 4)  (map layer image caching and map canvas
compositing), I was hoping that someone could point me towards some
documentation about this apparent feature.  Does anyone know any more about
this one?  Although I can just give it a try, it would be nice to know what
I'm getting in to before I get too far down the road.

Thanks,

Tom

On Wed, Nov 20, 2013 at 1:43 PM, Angus Carr <angus.carr at gmail.com> wrote:

> Hi, Tom. Your PyQT plugin would stall the main thread with a long process.
> You can launch threads, but that would still cause problems because of the
> python interpreter lock, which only allows a single thread access at once,
> so stall other threads.
>
> You will need to launch a process to calculate the results.
>
> There are mechanisms in python to chat between the processes in shared
> memory. I've played with some of them, and they can vary between fast and
> slow, depending on the details- it can still lock your main thread if you
> do it wrong.
>
> I had a multi-process app filling four cores with analysis, but it would
> slow down once the individual runs started bringing data back to the
> central collator. It would block the queue of all process, bringing
> performance to a crawl.
>
> If you are doing what I hope you are doing, then I would love to
> participate in bringing an FBP fire simulator to QGIS in open code.
> Particularly if it can then be used to do crazy things like be a process
> step in a forest simulation...
>
> Yes, I am a fellow Canadian QGIS user, and I believe we have met, although
> it's been a while... :-). I think we had a beer in Montana's in Thunder Bay
> in 2008, if that helps... :-)
>
> If you were interesting in collaborating, drop me a direct email. As long
> as it is open source, I can play. If it's tight to your company, I will
> have to defer.
>
> Cheers,
> Angus Carr.
>
>
>
>
>
> On Wed, Nov 20, 2013 at 1:01 PM, Tom Moore <moortom at gmail.com> wrote:
>
>> Greetings everyone!  I am new to the list and the QGIS community, and I
>> am looking for a bit of strategic direction from the experts.  I am
>> building a landscape-level wildfire simulation model.  The model will
>> simulate change and development of a natural forested landscape over
>> centuries of time as forest grow and wildfires occur.  The mechanics of the
>> model are fairly simple, and I will likely implement these in a c module
>> for reasonable performance.  The model simulates wildfire ignition and
>> spread on a raster surface, with generations representing future time
>> periods.  As the model runs it will gather statistics and save information
>> about conditions of each generation.
>>
>> I am considering using QGIS as a platform for the simulation model.  I
>> think there would be some nice benefits of making my model be a
>> python-based plugin that could leverage a lot of the data handling involved
>> with the spatial data sets, plus it looks like I could build an attractive
>> and function ui quite painlessly.  Here are some of the features that I
>> would like that I am pretty sure I can get from the QGIS platform:
>> - easy access to raster data sets via gdal.
>> - easy way to capture input parameters (data sources, regions of
>> interest, simulation model options) using pyQt and qgis platfrom.
>> - nice way to display final results from simulation, by save results out
>> to raster data set and then displaying in map canvas
>> - quite likely easy ways to display summary results as charts and
>> histograms.
>> So far +++ for QGIS.
>>
>> I have done a little bit of research to find out what it will take, and I
>> think the pieces are there. But there are a couple of things that I am not
>> sure about and I would like to ask the experts before I dig myself in to a
>> hole.
>>
>> The simulation model will take a long time to run and I don't want it to
>> stall the GUI as it is performing.
>> 1) Does qg support background processing?  Where can I read about the
>> synchronization mechanism?  If not, what is the best way to handle this?
>>
>> It would be really nice to be able to have a dynamic display of the
>> ongoing simulation as it is running.  I was thinking that I could use a
>> GDAL in-memory driver (http://www.gdal.org/frmt_mem.html) to hold
>> simulated conditions.
>> 2) If I did this, could I display this type of raster in the qg map
>> canvas?  What kind of synchronization is required so the simulation pauses
>> when the map is redrawn due to user interaction (zooming or panning for
>> example).
>>
>> 3) How will I be able to signal the map canvas when it is time for an
>> update?  Is QgsMapLayer.dataChanged() sufficient?   I see that someone else
>> has tried this and had difficulty:
>> http://article.gmane.org/gmane.comp.gis.qgis.devel/21481.  I assume that
>> the problem there was that the method was stalling the event loop.  What
>> are the rules for event loop interactions?
>>
>> 4) Can I update my raster map layer without having to re-draw the other
>> layers that may be on the the map canvas?  Does the map canvas support
>> compositing?  I see that QgsMapLayer supports an image cache, but I have
>> not found any explanation of how to use it.
>>
>> At this stage I'm looking for some clues.  I think there may be some
>> multi-threading issues to my design that require careful attention.  I see
>> that multi threading has been discussed in the context of rendering (
>> http://hub.qgis.org/issues/2037) but I am not sure about the current
>> status, or if this is the solution to my problem (perhaps not).  Any
>> suggestions, opinions or pointers to code or documentation would be greatly
>> appreciated.  If this is just a case of rtfm then I would be happy to hear
>> that also.
>>
>> Cheers,
>>
>> Tom
>>
>> _______________________________________________
>> Qgis-developer mailing list
>> Qgis-developer at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20131120/48a09000/attachment.html>