[QGIS-Developer] Multiprocessing QGIS

[Florian El Ahdab]

Hi.

I am posting my reply again because I missed the qgis-developer mail list
first...

It is indeed possible to achieve multiprocessing in QGIS.

But you should rely on Qt multi thread mechanisms rather than on the python
multiprocessing module.

Qgis is written in C++, based on the Qt library. Using the Qt
possibilities, even if it is thru python, is less likely to trigger
problems...

QThread is your friend here.
You can also use classes that are designed in QGis to ease things up, like
QgsTask (https://nocache.qgis.org/api/3.4/classQgsTask.html).

Regards
Florian.

Le sam. 27 mars 2021 à 13:48, Joao Folgado <jfolgado94 at gmail.com> a écrit :

> Hi guys,
>
> First of all thank you but, David Marteua and Sebastian Ernst, for the
> answer.
> So is impossible to use multiprocessing in Qgis plugin?
> This is a enormous limitations because Qgis have a lot of data  that
>  needs to be evaluated in same cases. Nowadays the cpu have 5,7, 9 cores
> and we can not use them in a plugin for example is bad. If some one have
> any idea who we can use multiprocessing in a plugin i will be thankful.
>
> Best regards,
> João Folgado
>
> David Marteau <dmarteau at 3liz.com> escreveu em sex., 26/03/2021 às 08:57 :
>
>> > I am not entirely sure what QGIS' (intended) behavior is in a case like
>> > this. From the top of my head, having read the relevant portion of its
>> > source code, I'd expect at least a second application window likely
>> > followed by non-deterministic behavior (again, depending on the
>> > use-case, likely resulting in a crash). Can someone elaborate?
>>
>> I can confirm that the result is non-deterministic even with headless
>> PyQgis programs.
>>
>> For programs relying heavily on multiprocessing the best strategy is to
>> use a `fork-server` process
>> started at the very beginning (before initializing Qgis) managing the
>> pool of child processes, so that each forked
>> process starts in the same state.
>>
>> David,
>>
>> Le 23/03/2021 à 19:22, Sebastian M. Ernst a écrit :
>> > Hi João, all,
>> >
>> > really interesting question.
>> >
>> >> however when i run the plugin a new instance of application is open
>> >> and give me an error.
>> > I assume that you are using `multiprocessing` [1] from Python's standard
>> > library. Its default behavior is to use process-based parallelism (as
>> > the title in the documentation says). I am assuming that you have not
>> > altered its default behavior (e.g. by configuring it to use threads
>> > instead). This is why you get at least another "worker" process if you
>> > e.g. create a process pool. I have successfully used this method with
>> > QGIS before, but always on Linux and OS X. I am just guessing that you
>> > are doing this on Windows, right?
>> >
>> > Depending on the operating system, `multiprocessing` uses different
>> > methods to create worker processes. On Linux and OS X, it simply forks
>> > [2] the main process. This way, you do not see a second application
>> > window popping up. You "simply" get a second, third, fourth, ... process
>> > "in the background". On Windows, the `fork` syscall does not exist,
>> > neither does something similar [3]. `multiprocessing` therefore starts a
>> > complete new process from scratch. If it was pure Python, it would
>> > simply start a new Python interpreter process. Because it is QGIS, it
>> > opens another instance of QGIS.
>> >
>> > I am not entirely sure what QGIS' (intended) behavior is in a case like
>> > this. From the top of my head, having read the relevant portion of its
>> > source code, I'd expect at least a second application window likely
>> > followed by non-deterministic behavior (again, depending on the
>> > use-case, likely resulting in a crash). Can someone elaborate?
>> >
>> > Best regards,
>> > Sebastian
>> >
>> >
>> > 1: https://docs.python.org/3/library/multiprocessing.html
>> > 2: https://en.wikipedia.org/wiki/Fork_(system_call)
>> > 3: https://stackoverflow.com/q/985281/1672565
>> >
>> >
>> > Am 23.03.21 um 13:08 schrieb Joao Folgado:
>> >> Hi everyone,
>> >>
>> >> I had created  a plugin for Qgis and I trying to optimize it. So i use
>> >> in my code the library python that have Queue whit multiprocessing
>> >> however when i run the plugin a new instance of application is open and
>> >> give me an error.
>> >> I had early use this type of multiprocessing before in simple projects
>> >> in university and it works ok.
>> >> Can someone help ? I researched in many forums, i see people whit the
>> >> same question but not with the answer.
>> >> Thank you very much.
>> >>
>> >> Best regards,
>> >> João Folgado
>> >> Portugal
>> >> --
>> >> JOÃO FOLGADO
>> >>
>> >> _______________________________________________
>> >> QGIS-Developer mailing list
>> >> QGIS-Developer at lists.osgeo.org
>> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> >>
>> > _______________________________________________
>> > QGIS-Developer mailing list
>> > QGIS-Developer at lists.osgeo.org
>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> _______________________________________________
>> QGIS-Developer mailing list
>> QGIS-Developer at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
> --
> JOÃO FOLGADO
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210328/84f82603/attachment.html>