[Qgis-developer] Multi threading and background processing

[Angus Carr]

Hi, Tom. Your PyQT plugin would stall the main thread with a long process.
You can launch threads, but that would still cause problems because of the
python interpreter lock, which only allows a single thread access at once,
so stall other threads.

You will need to launch a process to calculate the results.

There are mechanisms in python to chat between the processes in shared
memory. I've played with some of them, and they can vary between fast and
slow, depending on the details- it can still lock your main thread if you
do it wrong.

I had a multi-process app filling four cores with analysis, but it would
slow down once the individual runs started bringing data back to the
central collator. It would block the queue of all process, bringing
performance to a crawl.

If you are doing what I hope you are doing, then I would love to
participate in bringing an FBP fire simulator to QGIS in open code.
Particularly if it can then be used to do crazy things like be a process
step in a forest simulation...

Yes, I am a fellow Canadian QGIS user, and I believe we have met, although
it's been a while... :-). I think we had a beer in Montana's in Thunder Bay
in 2008, if that helps... :-)

If you were interesting in collaborating, drop me a direct email. As long
as it is open source, I can play. If it's tight to your company, I will
have to defer.

Cheers,
Angus Carr.





On Wed, Nov 20, 2013 at 1:01 PM, Tom Moore <moortom at gmail.com> wrote:

> Greetings everyone!  I am new to the list and the QGIS community, and I am
> looking for a bit of strategic direction from the experts.  I am building a
> landscape-level wildfire simulation model.  The model will simulate change
> and development of a natural forested landscape over centuries of time as
> forest grow and wildfires occur.  The mechanics of the model are fairly
> simple, and I will likely implement these in a c module for reasonable
> performance.  The model simulates wildfire ignition and spread on a raster
> surface, with generations representing future time periods.  As the model
> runs it will gather statistics and save information about conditions of
> each generation.
>
> I am considering using QGIS as a platform for the simulation model.  I
> think there would be some nice benefits of making my model be a
> python-based plugin that could leverage a lot of the data handling involved
> with the spatial data sets, plus it looks like I could build an attractive
> and function ui quite painlessly.  Here are some of the features that I
> would like that I am pretty sure I can get from the QGIS platform:
> - easy access to raster data sets via gdal.
> - easy way to capture input parameters (data sources, regions of interest,
> simulation model options) using pyQt and qgis platfrom.
> - nice way to display final results from simulation, by save results out
> to raster data set and then displaying in map canvas
> - quite likely easy ways to display summary results as charts and
> histograms.
> So far +++ for QGIS.
>
> I have done a little bit of research to find out what it will take, and I
> think the pieces are there. But there are a couple of things that I am not
> sure about and I would like to ask the experts before I dig myself in to a
> hole.
>
> The simulation model will take a long time to run and I don't want it to
> stall the GUI as it is performing.
> 1) Does qg support background processing?  Where can I read about the
> synchronization mechanism?  If not, what is the best way to handle this?
>
> It would be really nice to be able to have a dynamic display of the
> ongoing simulation as it is running.  I was thinking that I could use a
> GDAL in-memory driver (http://www.gdal.org/frmt_mem.html) to hold
> simulated conditions.
> 2) If I did this, could I display this type of raster in the qg map
> canvas?  What kind of synchronization is required so the simulation pauses
> when the map is redrawn due to user interaction (zooming or panning for
> example).
>
> 3) How will I be able to signal the map canvas when it is time for an
> update?  Is QgsMapLayer.dataChanged() sufficient?   I see that someone else
> has tried this and had difficulty:
> http://article.gmane.org/gmane.comp.gis.qgis.devel/21481.  I assume that
> the problem there was that the method was stalling the event loop.  What
> are the rules for event loop interactions?
>
> 4) Can I update my raster map layer without having to re-draw the other
> layers that may be on the the map canvas?  Does the map canvas support
> compositing?  I see that QgsMapLayer supports an image cache, but I have
> not found any explanation of how to use it.
>
> At this stage I'm looking for some clues.  I think there may be some
> multi-threading issues to my design that require careful attention.  I see
> that multi threading has been discussed in the context of rendering (
> http://hub.qgis.org/issues/2037) but I am not sure about the current
> status, or if this is the solution to my problem (perhaps not).  Any
> suggestions, opinions or pointers to code or documentation would be greatly
> appreciated.  If this is just a case of rtfm then I would be happy to hear
> that also.
>
> Cheers,
>
> Tom
>
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20131120/d7ecdd7b/attachment.html>