[GRASS-dev] multiprocessing problem

[Moritz Lennert]

Hi Luca,

Just two brainstorming ideas:

- From a rapid glance at the code it seems to me that you create a separate worker for each row in the raster. Correct ? AFAIR, spawning workers does create quite a bit of overhead. Depending on the row to column ratio of your raster, maybe you would be better off sending larger chunks to workers ?

- Depending on the number of parallel jobs, disk access can quickly become the bottleneck on non parallelized file systems. So it would be interesting to see if using fewer processes might actually speed up things. Then it is a question of finding the equilibrium.

Moritz 

Le 8 avril 2022 07:32:37 GMT+02:00, Luca Delucchi <lucadeluge at gmail.com> a écrit :
>Hi devs,
>
>I wrote an addons to calculate Rao's Q diversity index [0], I would
>like to speed it up using multiprocessing but with multiprocessing it
>took 2/3 times longer than a single process. Could someone look at the
>code and explain to me what I'm doing wrong?
>A colleague of mine suggested using cython with prange function [1]
>but I would avoid it since I need to study it and how to compile it in
>GRASS and I have no time to spend on it.
>
>thanks a lot
>Luca
>
>[0] https://github.com/lucadelu/grass-addons/tree/raoq/src/raster/r.raoq.area
>[1] https://cython.readthedocs.io/en/latest/src/userguide/parallelism.html
>