[QGIS-Developer] Multiprocessing QGIS
Pedro Camargo
c at margo.co
Sun Mar 28 15:59:59 PDT 2021
Florian,
I see that this is all very well developed in the C++ world, but I would like to add one question and one comment.
Question:
- Is there anything similar to Python Multiprocessing that can be used? Sometimes releasing the GIL is hard (requires going to C++/C/Cython). The original question may have been on this, I guess.
- Using Python threads works well when you do everything in Cython (for example) and release the GIL
Cheers,
Pedro
---- On Mon, 29 Mar 2021 05:00:01 +1000 <qgis-developer-request at lists.osgeo.org> wrote ----
> Send QGIS-Developer mailing list submissions to
> qgis-developer at lists.osgeo.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.osgeo.org/mailman/listinfo/qgis-developer
> or, via email, send a message with subject or body 'help' to
> qgis-developer-request at lists.osgeo.org
>
> You can reach the person managing the list at
> qgis-developer-owner at lists.osgeo.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of QGIS-Developer digest..."
>
>
> Today's Topics:
>
> 1. Re: Multiprocessing QGIS (Florian El Ahdab)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sun, 28 Mar 2021 16:48:12 +0200
> From: Florian El Ahdab <felahdab at gmail.com>
> To: Joao Folgado <jfolgado94 at gmail.com>
> Cc: David Marteau <dmarteau at 3liz.com>, qgis-developer
> <qgis-developer at lists.osgeo.org>
> Subject: Re: [QGIS-Developer] Multiprocessing QGIS
> Message-ID:
> <CAEeudyS4cjKzb9LTRjwyj7fovKgZpNO_hdw66eijgQXbwo61mQ at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi.
>
> I am posting my reply again because I missed the qgis-developer mail list
> first...
>
> It is indeed possible to achieve multiprocessing in QGIS.
>
> But you should rely on Qt multi thread mechanisms rather than on the python
> multiprocessing module.
>
> Qgis is written in C++, based on the Qt library. Using the Qt
> possibilities, even if it is thru python, is less likely to trigger
> problems...
>
> QThread is your friend here.
> You can also use classes that are designed in QGis to ease things up, like
> QgsTask (https://nocache.qgis.org/api/3.4/classQgsTask.html).
>
> Regards
> Florian.
>
> Le sam. 27 mars 2021 ? 13:48, Joao Folgado <jfolgado94 at gmail.com> a ?crit :
>
> > Hi guys,
> >
> > First of all thank you but, David Marteua and Sebastian Ernst, for the
> > answer.
> > So is impossible to use multiprocessing in Qgis plugin?
> > This is a enormous limitations because Qgis have a lot of data that
> > needs to be evaluated in same cases. Nowadays the cpu have 5,7, 9 cores
> > and we can not use them in a plugin for example is bad. If some one have
> > any idea who we can use multiprocessing in a plugin i will be thankful.
> >
> > Best regards,
> > Jo?o Folgado
> >
> > David Marteau <dmarteau at 3liz.com> escreveu em sex., 26/03/2021 ?s 08:57 :
> >
> >> > I am not entirely sure what QGIS' (intended) behavior is in a case like
> >> > this. From the top of my head, having read the relevant portion of its
> >> > source code, I'd expect at least a second application window likely
> >> > followed by non-deterministic behavior (again, depending on the
> >> > use-case, likely resulting in a crash). Can someone elaborate?
> >>
> >> I can confirm that the result is non-deterministic even with headless
> >> PyQgis programs.
> >>
> >> For programs relying heavily on multiprocessing the best strategy is to
> >> use a `fork-server` process
> >> started at the very beginning (before initializing Qgis) managing the
> >> pool of child processes, so that each forked
> >> process starts in the same state.
> >>
> >> David,
> >>
> >> Le 23/03/2021 ? 19:22, Sebastian M. Ernst a ?crit :
> >> > Hi Jo?o, all,
> >> >
> >> > really interesting question.
> >> >
> >> >> however when i run the plugin a new instance of application is open
> >> >> and give me an error.
> >> > I assume that you are using `multiprocessing` [1] from Python's standard
> >> > library. Its default behavior is to use process-based parallelism (as
> >> > the title in the documentation says). I am assuming that you have not
> >> > altered its default behavior (e.g. by configuring it to use threads
> >> > instead). This is why you get at least another "worker" process if you
> >> > e.g. create a process pool. I have successfully used this method with
> >> > QGIS before, but always on Linux and OS X. I am just guessing that you
> >> > are doing this on Windows, right?
> >> >
> >> > Depending on the operating system, `multiprocessing` uses different
> >> > methods to create worker processes. On Linux and OS X, it simply forks
> >> > [2] the main process. This way, you do not see a second application
> >> > window popping up. You "simply" get a second, third, fourth, ... process
> >> > "in the background". On Windows, the `fork` syscall does not exist,
> >> > neither does something similar [3]. `multiprocessing` therefore starts a
> >> > complete new process from scratch. If it was pure Python, it would
> >> > simply start a new Python interpreter process. Because it is QGIS, it
> >> > opens another instance of QGIS.
> >> >
> >> > I am not entirely sure what QGIS' (intended) behavior is in a case like
> >> > this. From the top of my head, having read the relevant portion of its
> >> > source code, I'd expect at least a second application window likely
> >> > followed by non-deterministic behavior (again, depending on the
> >> > use-case, likely resulting in a crash). Can someone elaborate?
> >> >
> >> > Best regards,
> >> > Sebastian
> >> >
> >> >
> >> > 1: https://docs.python.org/3/library/multiprocessing.html
> >> > 2: https://en.wikipedia.org/wiki/Fork_(system_call)
> >> > 3: https://stackoverflow.com/q/985281/1672565
> >> >
> >> >
> >> > Am 23.03.21 um 13:08 schrieb Joao Folgado:
> >> >> Hi everyone,
> >> >>
> >> >> I had created a plugin for Qgis and I trying to optimize it. So i use
> >> >> in my code the library python that have Queue whit multiprocessing
> >> >> however when i run the plugin a new instance of application is open and
> >> >> give me an error.
> >> >> I had early use this type of multiprocessing before in simple projects
> >> >> in university and it works ok.
> >> >> Can someone help ? I researched in many forums, i see people whit the
> >> >> same question but not with the answer.
> >> >> Thank you very much.
> >> >>
> >> >> Best regards,
> >> >> Jo?o Folgado
> >> >> Portugal
> >> >> --
> >> >> JO?O FOLGADO
> >> >>
> >> >> _______________________________________________
> >> >> QGIS-Developer mailing list
> >> >> QGIS-Developer at lists.osgeo.org
> >> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> >>
> >> > _______________________________________________
> >> > QGIS-Developer mailing list
> >> > QGIS-Developer at lists.osgeo.org
> >> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> _______________________________________________
> >> QGIS-Developer mailing list
> >> QGIS-Developer at lists.osgeo.org
> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >>
> > --
> > JO?O FOLGADO
> > _______________________________________________
> > QGIS-Developer mailing list
> > QGIS-Developer at lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210328/84f82603/attachment-0001.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
> ------------------------------
>
> End of QGIS-Developer Digest, Vol 185, Issue 60
> ***********************************************
>
More information about the QGIS-Developer
mailing list