[QGIS-Developer] Running grass algorithms in threads
Rudi von Staden
rudivs at gmail.com
Wed Aug 15 02:53:51 PDT 2018
Thanks all for your thoughts on this. I suspect the problem may be race
conditions being set up between
https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/grass7/Grass7Utils.py
and
https://github.com/qgis/QGIS/blob/master/python/plugins/processing/algs/grass7/Grass7Algorithm.py.
It attempts to use an existing session and shared mapset which could be
problematic from the sounds of things. It might be an idea to add an
optional parameter to grass algorithms to not use shared mapsets and create
a new mapset for each process?
The Grass7 log output from my script is below. The pairs of input rasters
for the parallel algorithms should be:
hm1.tif, LS90.tif
hm2.tif, LS14.tif
hm3.tif, HS90.tif
hm4.tif, LS90.tif
If could be that not all of the grass commands were logged, but it looks
like HS90.tif and LS14.tif were never passed as parameters to grass. I also
added a typical log when the algorithms are executed sequentially (3800-25.tif
is reused instead of hm1-4).
Kind regards,
Rudi
~~~ [ executed in parallel ] ~~~
2018-08-15T10:09:00 INFO processInputs end. Commands: ['g.proj -c
proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0
+datum=WGS84 +units=m +no_defs"', 'r.external
input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\cb4c839fbd6f4419873dbc29ffa2b973\\hm4.tif"
band=1 output="rast_5b73df9c811b22" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS14.tif"
band=1 output="rast_5b73df9c811b23" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.0757893076434']
2018-08-15T10:09:00 INFO processCommands end. Commands: ['g.proj -c
proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0
+datum=WGS84 +units=m +no_defs"', 'r.external
input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\cb4c839fbd6f4419873dbc29ffa2b973\\hm4.tif"
band=1 output="rast_5b73df9c811b22" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS14.tif"
band=1 output="rast_5b73df9c811b23" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.0757893076434', 'r.stats.zonal
base=rast_5b73df9c811b22 cover=rast_5b73df9c811b23 method="average"
output=output87eadbc7948b4b5990773af2261f0db8 --overwrite']
2018-08-15T10:09:00 INFO processInputs end. Commands: ['r.external
input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\79ea35021d26468c8220582c6397dd18\\hm1.tif"
band=1 output="rast_5b73df9c865914" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS90.tif"
band=1 output="rast_5b73df9c865915" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.0757893076434']
2018-08-15T10:09:00 INFO processCommands end. Commands: ['r.external
input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\79ea35021d26468c8220582c6397dd18\\hm1.tif"
band=1 output="rast_5b73df9c865914" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS90.tif"
band=1 output="rast_5b73df9c865915" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.0757893076434', 'r.stats.zonal
base=rast_5b73df9c865914 cover=rast_5b73df9c865915 method="average"
output=output8a381f2f7cc44b4eae194a33b1839457 --overwrite']
2018-08-15T10:09:00 INFO processInputs end. Commands: ['r.external
input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\1d1e057229aa4ee88a8abcb009b78f41\\hm3.tif"
band=1 output="rast_5b73df9c8ac116" --overwrite -o', 'g.region
n=-3680453.4502051217 s=-3763597.8879499235 e=-293721.76789138245
w=-478149.5738745922 res=92.0757893076434']
2018-08-15T10:09:00 INFO processCommands end. Commands: ['r.external
input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\1d1e057229aa4ee88a8abcb009b78f41\\hm3.tif"
band=1 output="rast_5b73df9c8ac116" --overwrite -o', 'g.region
n=-3680453.4502051217 s=-3763597.8879499235 e=-293721.76789138245
w=-478149.5738745922 res=92.0757893076434', 'r.stats.zonal
base=rast_5b73df9c8ac116 cover=rast_5b73df9c865915 method="average"
output=outputbef87f5fb08c43cd9e1cdd7ad44e5f1b --overwrite']
2018-08-15T10:09:00 INFO processInputs end. Commands: ['r.external
input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\4895f2e889a84f01aa852e7ca0035294\\hm2.tif"
band=1 output="rast_5b73df9c8defa7" --overwrite -o', 'g.region
n=-3680453.4502051217 s=-3763597.8879499235 e=-293721.76789138245
w=-478149.5738745922 res=92.0757893076434']
2018-08-15T10:09:00 INFO processCommands end. Commands: ['r.external
input="C:\\Users\\rudi\\AppData\\Local\\Temp\\processing_74832af828bb48a9bb0e64fd59d74f24\\4895f2e889a84f01aa852e7ca0035294\\hm2.tif"
band=1 output="rast_5b73df9c8defa7" --overwrite -o', 'g.region
n=-3680453.4502051217 s=-3763597.8879499235 e=-293721.76789138245
w=-478149.5738745922 res=92.0757893076434', 'r.stats.zonal
base=rast_5b73df9c8defa7 cover=rast_5b73df9c865915 method="average"
output=output349369bf07e14011bfd7611868151ec0 --overwrite']
~~~ [ executed sequentially ] ~~~
2018-08-15T11:46:27 INFO processInputs end. Commands: ['g.proj -c
proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0
+datum=WGS84 +units=m +no_defs"', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif"
band=1 output="rast_5b73f673681b58" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS90.tif"
band=1 output="rast_5b73f673681b59" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.07578930764342']
2018-08-15T11:46:27 INFO processCommands end. Commands: ['g.proj -c
proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0
+datum=WGS84 +units=m +no_defs"', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif"
band=1 output="rast_5b73f673681b58" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS90.tif"
band=1 output="rast_5b73f673681b59" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.07578930764342', 'r.stats.zonal
base=rast_5b73f673681b58 cover=rast_5b73f673681b59 method="average"
output=output64f55d9cf1ab4a95aa8bedf69f766783 --overwrite']
2018-08-15T11:46:58 INFO processInputs end. Commands: ['g.proj -c
proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0
+datum=WGS84 +units=m +no_defs"', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif"
band=1 output="rast_5b73f692c345310" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS14.tif"
band=1 output="rast_5b73f692c345311" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.07578930764342']
2018-08-15T11:46:58 INFO processCommands end. Commands: ['g.proj -c
proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0
+datum=WGS84 +units=m +no_defs"', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif"
band=1 output="rast_5b73f692c345310" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\LS14.tif"
band=1 output="rast_5b73f692c345311" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.07578930764342', 'r.stats.zonal
base=rast_5b73f692c345310 cover=rast_5b73f692c345311 method="average"
output=output1b75839b9455442e95ae87bddfe6c977 --overwrite']
2018-08-15T11:47:30 INFO processInputs end. Commands: ['g.proj -c
proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0
+datum=WGS84 +units=m +no_defs"', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif"
band=1 output="rast_5b73f6b20afa112" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS90.tif"
band=1 output="rast_5b73f6b20afa113" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.07578930764342']
2018-08-15T11:47:30 INFO processCommands end. Commands: ['g.proj -c
proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0
+datum=WGS84 +units=m +no_defs"', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif"
band=1 output="rast_5b73f6b20afa112" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS90.tif"
band=1 output="rast_5b73f6b20afa113" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.07578930764342', 'r.stats.zonal
base=rast_5b73f6b20afa112 cover=rast_5b73f6b20afa113 method="average"
output=output2c6300f21c524e0da74e65776d6292cf --overwrite']
2018-08-15T11:48:01 INFO processInputs end. Commands: ['g.proj -c
proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0
+datum=WGS84 +units=m +no_defs"', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif"
band=1 output="rast_5b73f6d1a6fa514" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS14.tif"
band=1 output="rast_5b73f6d1a6fa515" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.07578930764342']
2018-08-15T11:48:01 INFO processCommands end. Commands: ['g.proj -c
proj4="+proj=aea +lat_1=-24 +lat_2=-32 +lat_0=0 +lon_0=24 +x_0=0 +y_0=0
+datum=WGS84 +units=m +no_defs"', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\HabitatModels\\3800-25.tif"
band=1 output="rast_5b73f6d1a6fa514" --overwrite -o', 'r.external
input="C:\\Users\\rudi\\GIS\\Projects\\LizeSuitableHabitatModel\\Analysis\\Data\\HS14.tif"
band=1 output="rast_5b73f6d1a6fa515" --overwrite -o', 'g.region
n=-2332657.7474449594 s=-3803200.178477332 e=984393.5909882308
w=-834195.3236270341 res=92.07578930764342', 'r.stats.zonal
base=rast_5b73f6d1a6fa514 cover=rast_5b73f6d1a6fa515 method="average"
output=output744c02ef470a4ddbb2dbaaedf1bda2b0 --overwrite']
On Wed, 15 Aug 2018 at 08:16, Stefan Blumentrath <Stefan.Blumentrath at nina.no>
wrote:
> Dear Rudi, Nyall,
>
> GRASS is being used on HPC systems for heavily parallelisation. So, in
> principle, the answer is yes, you can for sure run GRASS algorithms in
> parallel.
> On Linux, I often run several commands in parallel using xargs. So it
> works just fine in many cases. GRASS also has some specific python
> functions for parallel processing. See also:
> https://grasswiki.osgeo.org/wiki/Parallel_GRASS_jobs
> https://grasswiki.osgeo.org/wiki/Parallelizing_Scripts
>
> However, if GRASS algorithms can be run in parallel in this particular
> case depends.
>
> E.g., if the algorithm in question temporarily modifies the computational
> region, parallel processes can get in the way for each other.
> Also, with SQLite as DB backend writing several vector maps (and attribute
> tables) in parallel will be a problem (due to SQLite locks).
>
> In addition, if GRASS commands can be executed in parallel in the QGIS
> Processing framework is probably yet another question, depending on how
> e.g. QGIS handles data management (locations and mapsets) esp. in more
> complex workflows / models...
>
> CCing also grass-dev list for more qualified answers...
>
> Cheers
> Stefan
>
>
> -----Original Message-----
> From: QGIS-Developer <qgis-developer-bounces at lists.osgeo.org> On Behalf
> Of Nyall Dawson
> Sent: onsdag 15. august 2018 01:10
> To: Rudi von Staden <rudivs at gmail.com>
> Cc: qgis-developer <qgis-developer at lists.osgeo.org>
> Subject: Re: [QGIS-Developer] Running grass algorithms in threads
>
> On Tue, 14 Aug 2018 at 21:43, Rudi von Staden <rudivs at gmail.com> wrote:
> >
> > Hi all,
> >
> > The bottleneck in my script at the moment is the calculation of zonal
> stats using 'grass7:r.stats.zonal'. I thought I might speed things up by
> using QgsTask.fromFunction() or QgsProcessingAlgRunnerTask() to run these
> calculations in parallel. In my tests of both approaches the tasks seem to
> complete (task.status() == QgsTask.Complete), but the output file is only
> generated for 1 of 4 parallel tasks (the task that finishes first).
> >
> > I'm assuming this is because grass algorithms are not thread safe? Or am
> I missing something in my implementation that could make this work?
>
> I strongly suspect that grass algorithms cannot be run in parallel.
> This is why they cannot run in the background in QGIS like the native/GDAL
> algorithms can. But I'd love for confirmation about this and whether
> there's any way to make GRASS multi-thread safe.
>
> Because this is grass related (and not QGIS specific) I'd suggest asking
> on the grass mailing list, and relaying any responses back here.
>
> Nyall
>
> >
> > Thanks,
> > Rudi
> >
> >
> >
> > My code for the QgsTask approach is as below:
> >
> > def getZonal(task, habitatModelFile, cover):
> > tempFile = QgsProcessingUtils.generateTempFilename("output.tif")
> > processing.run("grass7:r.stats.zonal", {
> > 'base':habitatModelFile,
> > 'cover':cover,
> > 'method':5,
> > '-c':False,
> > '-r':False,
> > 'output':tempFile,
> > 'GRASS_REGION_PARAMETER':None,
> > 'GRASS_REGION_CELLSIZE_PARAMETER':0,
> > 'GRASS_RASTER_FORMAT_OPT':'',
> >
> > 'GRASS_RASTER_FORMAT_META':''},context=context,feedback=algFeedback)
> >
> > if task.isCanceled():
> > deleteFile(tempFile)
> > return
> >
> > return tempFile
> >
> > ls90Task = QgsTask.fromFunction('LS90', getZonal,
> > habitatModelFile=hm1, cover=ls90Layer)
> > QgsApplication.taskManager().addTask(ls90Task)
> > feedback.pushInfo("Calculating LS14 mean...") ls14Task =
> > QgsTask.fromFunction('LS14 ', getZonal, habitatModelFile=hm2,
> > cover=ls14Layer)
> > QgsApplication.taskManager().addTask(ls14Task)
> > hs90Task = QgsTask.fromFunction('HS90 ', getZonal,
> > habitatModelFile=hm3, cover=hs90Layer)
> > QgsApplication.taskManager().addTask(hs90Task)
> > hs14Task = QgsTask.fromFunction('HS14 ', getZonal,
> > habitatModelFile=hm4, cover=hs14Layer)
> > QgsApplication.taskManager().addTask(hs14Task)
> >
> > while (len([t for t in [ls90Task.status(), ls14Task.status(),
> hs90Task.status(),
> > hs14Task.status()] if t in [QgsTask.Running,
> QgsTask.Queued]]) > 0)
> > and not feedback.isCanceled():
> > sleep(1)
> >
> > if feedback.isCanceled():
> > # some cleanup code (send task.cancel() and wait for tasks to
> terminate)
> > break
> >
> > ls90Result = ls90Task.returned_values
> > ls14Result = ls14Task.returned_values
> > hs90Result = hs90Task.returned_values # only this file exists
> > hs14Result = hs14Task.returned_values
> >
> >
> > _______________________________________________
> > QGIS-Developer mailing list
> > QGIS-Developer at lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20180815/bcbfd9b4/attachment-0001.html>
More information about the QGIS-Developer
mailing list