[QGIS-Developer] Multiprocessing QGIS

[Joao Folgado]

Hi guys,

First of all thank you but, David Marteua and Sebastian Ernst, for the
answer.
So is impossible to use multiprocessing in Qgis plugin?
This is a enormous limitations because Qgis have a lot of data  that  needs
to be evaluated in same cases. Nowadays the cpu have 5,7, 9 cores and we
can not use them in a plugin for example is bad. If some one have any idea
who we can use multiprocessing in a plugin i will be thankful.

Best regards,
João Folgado

David Marteau <dmarteau at 3liz.com> escreveu em sex., 26/03/2021 às 08:57 :

> > I am not entirely sure what QGIS' (intended) behavior is in a case like
> > this. From the top of my head, having read the relevant portion of its
> > source code, I'd expect at least a second application window likely
> > followed by non-deterministic behavior (again, depending on the
> > use-case, likely resulting in a crash). Can someone elaborate?
>
> I can confirm that the result is non-deterministic even with headless
> PyQgis programs.
>
> For programs relying heavily on multiprocessing the best strategy is to
> use a `fork-server` process
> started at the very beginning (before initializing Qgis) managing the
> pool of child processes, so that each forked
> process starts in the same state.
>
> David,
>
> Le 23/03/2021 à 19:22, Sebastian M. Ernst a écrit :
> > Hi João, all,
> >
> > really interesting question.
> >
> >> however when i run the plugin a new instance of application is open
> >> and give me an error.
> > I assume that you are using `multiprocessing` [1] from Python's standard
> > library. Its default behavior is to use process-based parallelism (as
> > the title in the documentation says). I am assuming that you have not
> > altered its default behavior (e.g. by configuring it to use threads
> > instead). This is why you get at least another "worker" process if you
> > e.g. create a process pool. I have successfully used this method with
> > QGIS before, but always on Linux and OS X. I am just guessing that you
> > are doing this on Windows, right?
> >
> > Depending on the operating system, `multiprocessing` uses different
> > methods to create worker processes. On Linux and OS X, it simply forks
> > [2] the main process. This way, you do not see a second application
> > window popping up. You "simply" get a second, third, fourth, ... process
> > "in the background". On Windows, the `fork` syscall does not exist,
> > neither does something similar [3]. `multiprocessing` therefore starts a
> > complete new process from scratch. If it was pure Python, it would
> > simply start a new Python interpreter process. Because it is QGIS, it
> > opens another instance of QGIS.
> >
> > I am not entirely sure what QGIS' (intended) behavior is in a case like
> > this. From the top of my head, having read the relevant portion of its
> > source code, I'd expect at least a second application window likely
> > followed by non-deterministic behavior (again, depending on the
> > use-case, likely resulting in a crash). Can someone elaborate?
> >
> > Best regards,
> > Sebastian
> >
> >
> > 1: https://docs.python.org/3/library/multiprocessing.html
> > 2: https://en.wikipedia.org/wiki/Fork_(system_call)
> > 3: https://stackoverflow.com/q/985281/1672565
> >
> >
> > Am 23.03.21 um 13:08 schrieb Joao Folgado:
> >> Hi everyone,
> >>
> >> I had created  a plugin for Qgis and I trying to optimize it. So i use
> >> in my code the library python that have Queue whit multiprocessing
> >> however when i run the plugin a new instance of application is open and
> >> give me an error.
> >> I had early use this type of multiprocessing before in simple projects
> >> in university and it works ok.
> >> Can someone help ? I researched in many forums, i see people whit the
> >> same question but not with the answer.
> >> Thank you very much.
> >>
> >> Best regards,
> >> João Folgado
> >> Portugal
> >> --
> >> JOÃO FOLGADO
> >>
> >> _______________________________________________
> >> QGIS-Developer mailing list
> >> QGIS-Developer at lists.osgeo.org
> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >>
> > _______________________________________________
> > QGIS-Developer mailing list
> > QGIS-Developer at lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
-- 
JOÃO FOLGADO
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210327/9397e2f1/attachment-0001.html>