[QGIS-Developer] Multiprocessing QGIS

[David Marteau]

> I am not entirely sure what QGIS' (intended) behavior is in a case like
> this. From the top of my head, having read the relevant portion of its
> source code, I'd expect at least a second application window likely
> followed by non-deterministic behavior (again, depending on the
> use-case, likely resulting in a crash). Can someone elaborate?

I can confirm that the result is non-deterministic even with headless 
PyQgis programs.

For programs relying heavily on multiprocessing the best strategy is to 
use a `fork-server` process
started at the very beginning (before initializing Qgis) managing the 
pool of child processes, so that each forked
process starts in the same state.

David,

Le 23/03/2021 à 19:22, Sebastian M. Ernst a écrit :
> Hi João, all,
>
> really interesting question.
>
>> however when i run the plugin a new instance of application is open
>> and give me an error.
> I assume that you are using `multiprocessing` [1] from Python's standard
> library. Its default behavior is to use process-based parallelism (as
> the title in the documentation says). I am assuming that you have not
> altered its default behavior (e.g. by configuring it to use threads
> instead). This is why you get at least another "worker" process if you
> e.g. create a process pool. I have successfully used this method with
> QGIS before, but always on Linux and OS X. I am just guessing that you
> are doing this on Windows, right?
>
> Depending on the operating system, `multiprocessing` uses different
> methods to create worker processes. On Linux and OS X, it simply forks
> [2] the main process. This way, you do not see a second application
> window popping up. You "simply" get a second, third, fourth, ... process
> "in the background". On Windows, the `fork` syscall does not exist,
> neither does something similar [3]. `multiprocessing` therefore starts a
> complete new process from scratch. If it was pure Python, it would
> simply start a new Python interpreter process. Because it is QGIS, it
> opens another instance of QGIS.
>
> I am not entirely sure what QGIS' (intended) behavior is in a case like
> this. From the top of my head, having read the relevant portion of its
> source code, I'd expect at least a second application window likely
> followed by non-deterministic behavior (again, depending on the
> use-case, likely resulting in a crash). Can someone elaborate?
>
> Best regards,
> Sebastian
>
>
> 1: https://docs.python.org/3/library/multiprocessing.html
> 2: https://en.wikipedia.org/wiki/Fork_(system_call)
> 3: https://stackoverflow.com/q/985281/1672565
>
>
> Am 23.03.21 um 13:08 schrieb Joao Folgado:
>> Hi everyone,
>>
>> I had created  a plugin for Qgis and I trying to optimize it. So i use
>> in my code the library python that have Queue whit multiprocessing
>> however when i run the plugin a new instance of application is open and
>> give me an error.
>> I had early use this type of multiprocessing before in simple projects
>> in university and it works ok.
>> Can someone help ? I researched in many forums, i see people whit the
>> same question but not with the answer.
>> Thank you very much.
>>
>> Best regards,
>> João Folgado
>> Portugal
>> -- 
>> JOÃO FOLGADO
>>
>> _______________________________________________
>> QGIS-Developer mailing list
>> QGIS-Developer at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer