[GRASS-dev] [GRASS GIS] #3166: Parallelization with tiling for grass.script

[GRASS GIS]

#3166: Parallelization with tiling for grass.script
--------------------------+------------------------------
  Reporter:  wenzeslaus   |      Owner:  grass-dev@…
      Type:  enhancement  |     Status:  new
  Priority:  normal       |  Milestone:  7.4.0
 Component:  Python       |    Version:  unspecified
Resolution:               |   Keywords:  script, parallel
       CPU:  Unspecified  |   Platform:  Unspecified
--------------------------+------------------------------

Comment (by wenzeslaus):

 Replying to [comment:1 mlennert]:
 > Wow, this looks great !

 Note also what is in PyGRASS. Especially after r69507.

 > Could you just explain the relation

 Partial duplication. As in the case of `grass.script` and
 `grass.pygrass.modules`.

 > difference between this and the GridModule in pygrass ?

 * This supports list of modules (workflow) which are executed subsequently
 on the given tile.
 * User needs to prepare the list in a for loop (as opposed to not using
 any for loop). This is because it is in fact derived from the non-tiled
 parallelization API which is more general, so the user in fact loops over
 what needs to be done (with or without parallelization).
 * Related to that, the API for simple parallel processing, parallel
 processing of series of maps, and tiled parallel processing of series of
 maps is the same.
 * User needs to "help" the functions and objects by providing the with
 some metadata, i.e. types and names of the maps to patch (for patching),
 because no module run metadata are available in `grass.script`.
 * Some of the execution details are lost, e.g. only last command's textual
 output is preserved.
 * `GridModule` uses separate mapsets for individual tiles, this uses
 `WIND_OVERRIDE`.
 * `GridModule` uses PyGRASS ctypes wrappers for patching, this
 (potentially huge) expression `r.mapcalc` and `r3.mapcalc`.
 * The interface is like `grass.script`, not like `grass.pygrass.modules`.
 * It is not complete.
 * The implementation is now 300 lines. `MultiModule` alone has 200.
 * It uses from multiprocessing `Pool.map_async` function (which may be the
 cause of problem with interrupting).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3166#comment:3>
GRASS GIS <https://grass.osgeo.org>