[GRASS-dev] [GRASS GIS] #3166: Parallelization with tiling for grass.script

[GRASS GIS]

#3166: Parallelization with tiling for grass.script
------------------------------+-------------------------
 Reporter:  wenzeslaus        |      Owner:  grass-dev@…
     Type:  enhancement       |     Status:  new
 Priority:  normal            |  Milestone:  7.4.0
Component:  Python            |    Version:  unspecified
 Keywords:  script, parallel  |        CPU:  Unspecified
 Platform:  Unspecified       |
------------------------------+-------------------------
 At the same time as r69507, I was working on a simpler approach based on
 `grass.script`. At this point, it can do tiling and patching of rasters
 and partially 3D rasters. Series of commands is executed on each tile.
 This and also the patching runs in parallel. The syntax is very similar to
 `grass.script` with a convenient function for tile naming (which is
 partially a user responsibility).

 Here is an tiling example:

 {{{
 #!python
 # this is the control object
 tiled_workflow = TiledWorkflow(nprocs=4, width=500, height=500,
 overlap=10)
 for namer, workflow in tiled_workflow:
     slope = namer.name('raster', 'slope')
     aspect = namer.name('raster','aspect')

     # now do all as usually, workflow is equivalent of `grass.script`
     workflow.run_command('r.slope.aspect', elevation='fractal_surface',
                          slope=slope)
     workflow.run_command('r.slope.aspect', elevation='fractal_surface',
                          aspect=aspect)
     workflow.parse_command('g.region', flags='pg')

 # nothing was actually done till now
 # do the parallel processing and patching
 results = tiled_workflow.execute()

 # iterate over the results (here from g.region)
 for result in results:
     for key, value in result.iteritems():
         print key, value
 }}}

 Example using much smaller portion of the API. Creates list of modules
 which are then executed in the background. When import of the `parallel`
 module fails, `grass.script` is used instead without any changed in the
 main part of the code.

 {{{
 #!python
 try:
     from grass.script.parallel import ModuleCallList, execute_by_module
     call = ModuleCallList()
     parallel = True
 except ImportError:
     call = gs  # fall back to grass.script
     parallel = False
 for i in range(map_min, map_max + 1):
     call.mapcalc(expr, num=i)
 if parallel:
     execute_by_module(call, nprocs=4)
 }}}

 The current code uses `r.mapcalc` for patching and PyGRASS code computing
 the tiles. One of the main issues with the current code is that it does
 not finish when there is an error in the executed module.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3166>
GRASS GIS <https://grass.osgeo.org>