[QGIS-Developer] Multiprocessing QGIS

Nyall Dawson nyall.dawson at gmail.com
Sun Mar 28 16:17:08 PDT 2021 

On Mon, 29 Mar 2021 at 09:00, Pedro Camargo <c at margo.co> wrote:
>
> Florian,
>
>               I see that this is all very well developed in the C++ world, but I would like to add one question and one comment.
>
> Question:
>
> - Is there anything similar to Python Multiprocessing that can be used?  Sometimes releasing the GIL is hard (requires going to C++/C/Cython).  The original question may have been on this, I guess.

As I've mentioned previously on this thread, this isn't an issue with
PyQGIS. The GIL is automatically released whenever you call a PyQGIS
method, unless it's absolutely certain that the method will be
near-instant to call. So it's quite straightforward to use
multiprocessing in PyQGIS via the Qt methods and do things like
calculating intersections for different objects across multiple
threads at once without having to worry about the GIL at all...

Nyall


>
> - Using Python threads works well when you do everything in Cython (for example) and release the GIL
>
> Cheers,
> Pedro
>
>
>  ---- On Mon, 29 Mar 2021 05:00:01 +1000  <qgis-developer-request at lists.osgeo.org> wrote ----
>  > Send QGIS-Developer mailing list submissions to
>  >     qgis-developer at lists.osgeo.org
>  >
>  > To subscribe or unsubscribe via the World Wide Web, visit
>  >     https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > or, via email, send a message with subject or body 'help' to
>  >     qgis-developer-request at lists.osgeo.org
>  >
>  > You can reach the person managing the list at
>  >     qgis-developer-owner at lists.osgeo.org
>  >
>  > When replying, please edit your Subject line so it is more specific
>  > than "Re: Contents of QGIS-Developer digest..."
>  >
>  >
>  > Today's Topics:
>  >
>  >    1. Re: Multiprocessing QGIS (Florian El Ahdab)
>  >
>  >
>  > ----------------------------------------------------------------------
>  >
>  > Message: 1
>  > Date: Sun, 28 Mar 2021 16:48:12 +0200
>  > From: Florian El Ahdab <felahdab at gmail.com>
>  > To: Joao Folgado <jfolgado94 at gmail.com>
>  > Cc: David Marteau <dmarteau at 3liz.com>, qgis-developer
>  >     <qgis-developer at lists.osgeo.org>
>  > Subject: Re: [QGIS-Developer] Multiprocessing QGIS
>  > Message-ID:
>  >     <CAEeudyS4cjKzb9LTRjwyj7fovKgZpNO_hdw66eijgQXbwo61mQ at mail.gmail.com>
>  > Content-Type: text/plain; charset="utf-8"
>  >
>  > Hi.
>  >
>  > I am posting my reply again because I missed the qgis-developer mail list
>  > first...
>  >
>  > It is indeed possible to achieve multiprocessing in QGIS.
>  >
>  > But you should rely on Qt multi thread mechanisms rather than on the python
>  > multiprocessing module.
>  >
>  > Qgis is written in C++, based on the Qt library. Using the Qt
>  > possibilities, even if it is thru python, is less likely to trigger
>  > problems...
>  >
>  > QThread is your friend here.
>  > You can also use classes that are designed in QGis to ease things up, like
>  > QgsTask (https://nocache.qgis.org/api/3.4/classQgsTask.html).
>  >
>  > Regards
>  > Florian.
>  >
>  > Le sam. 27 mars 2021 ? 13:48, Joao Folgado <jfolgado94 at gmail.com> a ?crit :
>  >
>  > > Hi guys,
>  > >
>  > > First of all thank you but, David Marteua and Sebastian Ernst, for the
>  > > answer.
>  > > So is impossible to use multiprocessing in Qgis plugin?
>  > > This is a enormous limitations because Qgis have a lot of data  that
>  > >  needs to be evaluated in same cases. Nowadays the cpu have 5,7, 9 cores
>  > > and we can not use them in a plugin for example is bad. If some one have
>  > > any idea who we can use multiprocessing in a plugin i will be thankful.
>  > >
>  > > Best regards,
>  > > Jo?o Folgado
>  > >
>  > > David Marteau <dmarteau at 3liz.com> escreveu em sex., 26/03/2021 ?s 08:57 :
>  > >
>  > >> > I am not entirely sure what QGIS' (intended) behavior is in a case like
>  > >> > this. From the top of my head, having read the relevant portion of its
>  > >> > source code, I'd expect at least a second application window likely
>  > >> > followed by non-deterministic behavior (again, depending on the
>  > >> > use-case, likely resulting in a crash). Can someone elaborate?
>  > >>
>  > >> I can confirm that the result is non-deterministic even with headless
>  > >> PyQgis programs.
>  > >>
>  > >> For programs relying heavily on multiprocessing the best strategy is to
>  > >> use a `fork-server` process
>  > >> started at the very beginning (before initializing Qgis) managing the
>  > >> pool of child processes, so that each forked
>  > >> process starts in the same state.
>  > >>
>  > >> David,
>  > >>
>  > >> Le 23/03/2021 ? 19:22, Sebastian M. Ernst a ?crit :
>  > >> > Hi Jo?o, all,
>  > >> >
>  > >> > really interesting question.
>  > >> >
>  > >> >> however when i run the plugin a new instance of application is open
>  > >> >> and give me an error.
>  > >> > I assume that you are using `multiprocessing` [1] from Python's standard
>  > >> > library. Its default behavior is to use process-based parallelism (as
>  > >> > the title in the documentation says). I am assuming that you have not
>  > >> > altered its default behavior (e.g. by configuring it to use threads
>  > >> > instead). This is why you get at least another "worker" process if you
>  > >> > e.g. create a process pool. I have successfully used this method with
>  > >> > QGIS before, but always on Linux and OS X. I am just guessing that you
>  > >> > are doing this on Windows, right?
>  > >> >
>  > >> > Depending on the operating system, `multiprocessing` uses different
>  > >> > methods to create worker processes. On Linux and OS X, it simply forks
>  > >> > [2] the main process. This way, you do not see a second application
>  > >> > window popping up. You "simply" get a second, third, fourth, ... process
>  > >> > "in the background". On Windows, the `fork` syscall does not exist,
>  > >> > neither does something similar [3]. `multiprocessing` therefore starts a
>  > >> > complete new process from scratch. If it was pure Python, it would
>  > >> > simply start a new Python interpreter process. Because it is QGIS, it
>  > >> > opens another instance of QGIS.
>  > >> >
>  > >> > I am not entirely sure what QGIS' (intended) behavior is in a case like
>  > >> > this. From the top of my head, having read the relevant portion of its
>  > >> > source code, I'd expect at least a second application window likely
>  > >> > followed by non-deterministic behavior (again, depending on the
>  > >> > use-case, likely resulting in a crash). Can someone elaborate?
>  > >> >
>  > >> > Best regards,
>  > >> > Sebastian
>  > >> >
>  > >> >
>  > >> > 1: https://docs.python.org/3/library/multiprocessing.html
>  > >> > 2: https://en.wikipedia.org/wiki/Fork_(system_call)
>  > >> > 3: https://stackoverflow.com/q/985281/1672565
>  > >> >
>  > >> >
>  > >> > Am 23.03.21 um 13:08 schrieb Joao Folgado:
>  > >> >> Hi everyone,
>  > >> >>
>  > >> >> I had created  a plugin for Qgis and I trying to optimize it. So i use
>  > >> >> in my code the library python that have Queue whit multiprocessing
>  > >> >> however when i run the plugin a new instance of application is open and
>  > >> >> give me an error.
>  > >> >> I had early use this type of multiprocessing before in simple projects
>  > >> >> in university and it works ok.
>  > >> >> Can someone help ? I researched in many forums, i see people whit the
>  > >> >> same question but not with the answer.
>  > >> >> Thank you very much.
>  > >> >>
>  > >> >> Best regards,
>  > >> >> Jo?o Folgado
>  > >> >> Portugal
>  > >> >> --
>  > >> >> JO?O FOLGADO
>  > >> >>
>  > >> >> _______________________________________________
>  > >> >> QGIS-Developer mailing list
>  > >> >> QGIS-Developer at lists.osgeo.org
>  > >> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > >> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > >> >>
>  > >> > _______________________________________________
>  > >> > QGIS-Developer mailing list
>  > >> > QGIS-Developer at lists.osgeo.org
>  > >> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > >> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > >> _______________________________________________
>  > >> QGIS-Developer mailing list
>  > >> QGIS-Developer at lists.osgeo.org
>  > >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > >>
>  > > --
>  > > JO?O FOLGADO
>  > > _______________________________________________
>  > > QGIS-Developer mailing list
>  > > QGIS-Developer at lists.osgeo.org
>  > > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > >
>  > -------------- next part --------------
>  > An HTML attachment was scrubbed...
>  > URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210328/84f82603/attachment-0001.html>
>  >
>  > ------------------------------
>  >
>  > Subject: Digest Footer
>  >
>  > _______________________________________________
>  > QGIS-Developer mailing list
>  > QGIS-Developer at lists.osgeo.org
>  > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>  >
>  >
>  > ------------------------------
>  >
>  > End of QGIS-Developer Digest, Vol 185, Issue 60
>  > ***********************************************
>  >
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

More information about the QGIS-Developer mailing list