[QGIS-Developer] Running grass algorithms in threads

[Rudi von Staden]

Hi all,

The bottleneck in my script at the moment is the calculation of zonal stats
using 'grass7:r.stats.zonal'. I thought I might speed things up by using
QgsTask.fromFunction() or QgsProcessingAlgRunnerTask() to run these
calculations in parallel. In my tests of both approaches the tasks seem to
complete (task.status() == QgsTask.Complete), but the output file is only
generated for 1 of 4 parallel tasks (the task that finishes first).

I'm assuming this is because grass algorithms are not thread safe? Or am I
missing something in my implementation that could make this work?

Thanks,
Rudi



My code for the QgsTask approach is as below:

def getZonal(task, habitatModelFile, cover):
    tempFile = QgsProcessingUtils.generateTempFilename("output.tif")
    processing.run("grass7:r.stats.zonal", {
        'base':habitatModelFile,
        'cover':cover,
        'method':5,
        '-c':False,
        '-r':False,
        'output':tempFile,
        'GRASS_REGION_PARAMETER':None,
        'GRASS_REGION_CELLSIZE_PARAMETER':0,
        'GRASS_RASTER_FORMAT_OPT':'',
        'GRASS_RASTER_FORMAT_META':''},context=context,feedback=algFeedback)

    if task.isCanceled():
        deleteFile(tempFile)
        return

    return tempFile

ls90Task = QgsTask.fromFunction('LS90', getZonal, habitatModelFile=hm1,
cover=ls90Layer)
QgsApplication.taskManager().addTask(ls90Task)
feedback.pushInfo("Calculating LS14 mean...")
ls14Task = QgsTask.fromFunction('LS14 ', getZonal, habitatModelFile=hm2,
cover=ls14Layer)
QgsApplication.taskManager().addTask(ls14Task)
hs90Task = QgsTask.fromFunction('HS90 ', getZonal, habitatModelFile=hm3,
cover=hs90Layer)
QgsApplication.taskManager().addTask(hs90Task)
hs14Task = QgsTask.fromFunction('HS14 ', getZonal, habitatModelFile=hm4,
cover=hs14Layer)
QgsApplication.taskManager().addTask(hs14Task)

while (len([t for t in [ls90Task.status(), ls14Task.status(),
hs90Task.status(),
            hs14Task.status()] if t in [QgsTask.Running, QgsTask.Queued]])
> 0)
            and not feedback.isCanceled():
    sleep(1)

if feedback.isCanceled():
    # some cleanup code (send task.cancel() and wait for tasks to terminate)
    break

ls90Result = ls90Task.returned_values
ls14Result = ls14Task.returned_values
hs90Result = hs90Task.returned_values   # only this file exists
hs14Result = hs14Task.returned_values
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20180814/533413c6/attachment.html>