<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 14, 2018 at 7:10 PM, Nyall Dawson <span dir="ltr"><<a href="mailto:nyall.dawson@gmail.com" target="_blank">nyall.dawson@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On Tue, 14 Aug 2018 at 21:43, Rudi von Staden <<a href="mailto:rudivs@gmail.com">rudivs@gmail.com</a>> wrote:<br>
><br>
> Hi all,<br>
><br>
> The bottleneck in my script at the moment is the calculation of zonal stats using 'grass7:r.stats.zonal'. I thought I might speed things up by using QgsTask.fromFunction() or QgsProcessingAlgRunnerTask() to run these calculations in parallel. In my tests of both approaches the tasks seem to complete (task.status() == QgsTask.Complete), but the output file is only generated for 1 of 4 parallel tasks (the task that finishes first).<br>
><br>
> I'm assuming this is because grass algorithms are not thread safe? Or am I missing something in my implementation that could make this work?<br>
<br>
</span>I strongly suspect that grass algorithms cannot be run in parallel.<br>
This is why they cannot run in the background in QGIS like the<br>
native/GDAL algorithms can. But I'd love for confirmation about this<br>
and whether there's any way to make GRASS multi-thread safe.<br></blockquote><div><br></div><div><br></div><div>In general, it works. You can run GRASS modules in parallel if you set things right which is best achieved by running the parallel processes in separate GRASS mapsets.<br></div><div><br></div><div>GRASS modules are separate processes, so we are talking about parallel processes, rather than threads, so there are pretty separated. You can run for example (assuming GRASS GIS session in NC SPM location, new/empty mapset, and Linux command line, so that & starts process in the background):<br></div><div><br></div><div>r.neighbors input=elevation output=elevation_1 size=21 &<br>r.neighbors input=elevation output=elevation_2 size=21 &<br>r.neighbors input=elevation output=elevation_3 size=21 &<br>r.neighbors input=elevation output=elevation_4 size=21 &<br></div><div><br></div><div>However, conflicts may arise if you are changing computational region at the same time as doing calculations or if you are writing vectors using the default settings for attribute table, i.e. one SQLite db for all vector maps in a mapset. You can make these things work, e.g. by passing a computational region through environment rather than by g.region or by using different backend for attributes. However, the safest way are the separate mapsets, for example (assuming Linux for & and an existing location called nc_spm):<br></div><div><br></div><div># create the mapsets<br></div><div>grass -e -c ~/grassdata/nc_spm/par1</div><div>grass -e -c ~/grassdata/nc_spm/par2<br></div><div>grass -e -c ~/grassdata/nc_spm/par3<br></div><div>grass -e -c ~/grassdata/nc_spm/par4</div><div># run v.random (just an example which creates vector with attributes)<br></div><div>grass ~/grassdata/nc_spm/par1 --exec v.random output=points_1 column=value npoints=1000000 &<br>grass ~/grassdata/nc_spm/par2 --exec v.random output=points_2 column=value npoints=1000000 &<br>grass ~/grassdata/nc_spm/par3 --exec v.random output=points_3 column=value npoints=1000000 &<br>grass ~/grassdata/nc_spm/par4 --exec v.random output=points_4 column=value npoints=1000000 &</div><div># just to finish the example, let's merge the vectors in a new mapset</div><div>grass -e -c ~/grassdata/nc_spm/par<br></div><div>grass ~/grassdata/nc_spm/par --exec v.patch input=points_1@par1,points_2@par2,points_3@par3,points_4@par4 output=points</div><div>grass ~/grassdata/nc_spm/par --exec <a href="http://v.info">v.info</a> map=points -t</div><div># and delete the rest</div><div>rm ~/grassdata/nc_spm/par1</div><div>rm ~/grassdata/nc_spm/par2</div><div>rm ~/grassdata/nc_spm/par3</div><div>rm ~/grassdata/nc_spm/par4<br></div><div></div><div><br></div><div>I'm assuming that we are talking about running algorithms in parallel in QGIS, not parallelism inside the algorithms. Other considerations apply to that (parallelization is controlled by the modules themselves, see e.g. nprocs option for r.sun or v.surf.rst in G 7.4). Note that I'm talking about (pure) GRASS, so it depends how QGIS is handling it (I recall it is using --exec, but I don't know what it is doing with locations and mapsets). Please also note that I didn't measure if the v.random example would be actually more advantageous the a single process.<br></div><div><br></div><div>Best,</div><div>Vaclav<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Because this is grass related (and not QGIS specific) I'd suggest<br>
asking on the grass mailing list, and relaying any responses back<br>
here.<br></blockquote><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Nyall<br>
<div><div class="gmail-h5"><br>
><br>
> Thanks,<br>
> Rudi<br>
><br>
><br>
><br>
> My code for the QgsTask approach is as below:<br>
><br>
> def getZonal(task, habitatModelFile, cover):<br>
> tempFile = QgsProcessingUtils.<wbr>generateTempFilename("output.<wbr>tif")<br>
> processing.run("grass7:r.<wbr>stats.zonal", {<br>
> 'base':habitatModelFile,<br>
> 'cover':cover,<br>
> 'method':5,<br>
> '-c':False,<br>
> '-r':False,<br>
> 'output':tempFile,<br>
> 'GRASS_REGION_PARAMETER':None,<br>
> 'GRASS_REGION_CELLSIZE_<wbr>PARAMETER':0,<br>
> 'GRASS_RASTER_FORMAT_OPT':'',<br>
> 'GRASS_RASTER_FORMAT_META':''}<wbr>,context=context,feedback=<wbr>algFeedback)<br>
><br>
> if task.isCanceled():<br>
> deleteFile(tempFile)<br>
> return<br>
><br>
> return tempFile<br>
><br>
> ls90Task = QgsTask.fromFunction('LS90', getZonal, habitatModelFile=hm1, cover=ls90Layer)<br>
> QgsApplication.taskManager().<wbr>addTask(ls90Task)<br>
> feedback.pushInfo("Calculating LS14 mean...")<br>
> ls14Task = QgsTask.fromFunction('LS14 ', getZonal, habitatModelFile=hm2, cover=ls14Layer)<br>
> QgsApplication.taskManager().<wbr>addTask(ls14Task)<br>
> hs90Task = QgsTask.fromFunction('HS90 ', getZonal, habitatModelFile=hm3, cover=hs90Layer)<br>
> QgsApplication.taskManager().<wbr>addTask(hs90Task)<br>
> hs14Task = QgsTask.fromFunction('HS14 ', getZonal, habitatModelFile=hm4, cover=hs14Layer)<br>
> QgsApplication.taskManager().<wbr>addTask(hs14Task)<br>
><br>
> while (len([t for t in [ls90Task.status(), ls14Task.status(), hs90Task.status(),<br>
> hs14Task.status()] if t in [QgsTask.Running, QgsTask.Queued]]) > 0)<br>
> and not feedback.isCanceled():<br>
> sleep(1)<br>
><br>
> if feedback.isCanceled():<br>
> # some cleanup code (send task.cancel() and wait for tasks to terminate)<br>
> break<br>
><br>
> ls90Result = ls90Task.returned_values<br>
> ls14Result = ls14Task.returned_values<br>
> hs90Result = hs90Task.returned_values # only this file exists<br>
> hs14Result = hs14Task.returned_values<br>
><br>
><br>
</div></div>> ______________________________<wbr>_________________<br>
> QGIS-Developer mailing list<br>
> <a href="mailto:QGIS-Developer@lists.osgeo.org">QGIS-Developer@lists.osgeo.org</a><br>
> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-<wbr>developer</a><br>
> Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-<wbr>developer</a><br>
______________________________<wbr>_________________<br>
QGIS-Developer mailing list<br>
<a href="mailto:QGIS-Developer@lists.osgeo.org">QGIS-Developer@lists.osgeo.org</a><br>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-<wbr>developer</a><br>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">https://lists.osgeo.org/<wbr>mailman/listinfo/qgis-<wbr>developer</a></blockquote></div><br></div></div>