[QGIS-Developer] Multiprocessing QGIS
Pedro Camargo
c at margo.co
Sun Mar 28 17:33:11 PDT 2021
Hi Nyall,
I have seen that, and that's why I prefaced with an observation that this seems to be handled pretty well in the C++ world inside QGIS.
However, there are methods/functions/algorithms that are not available in QGIS and that would benefit from being run in multiple threads (if the user knows how to go down to C/C++ and release the GIL themselves) or to just brute-force it with Multiprocessing.
One might ask why is somebody doing it inside QGIS, then? Although that is a fair question, I would say that this is a testament to how good the QGIS interface and tools are, as people want to do things within its boundaries.
AequilibraE itself, which is a package for transportation modelling, takes its cues from what is available in the QGIS packaging for Windows so it can be used by non-programmers and has all its core functionality based solely on the packages available by default (some functionality depends on the user installing further packages).
So, although it is fair to have a stance that QGIS is NOT intended to be a virtual environment that comes with great GIS software, I would say it is understandable that some people would like it to be.
Cheers,
Pedro
PS - And I promise I won't cause any more steer (this week), Nyall! :-D
---- On Mon, 29 Mar 2021 09:17:08 +1000 Nyall Dawson <nyall.dawson at gmail.com> wrote ----
> On Mon, 29 Mar 2021 at 09:00, Pedro Camargo <c at margo.co> wrote:
> >
> > Florian,
> >
> > I see that this is all very well developed in the C++ world, but I would like to add one question and one comment.
> >
> > Question:
> >
> > - Is there anything similar to Python Multiprocessing that can be used? Sometimes releasing the GIL is hard (requires going to C++/C/Cython). The original question may have been on this, I guess.
>
> As I've mentioned previously on this thread, this isn't an issue with
> PyQGIS. The GIL is automatically released whenever you call a PyQGIS
> method, unless it's absolutely certain that the method will be
> near-instant to call. So it's quite straightforward to use
> multiprocessing in PyQGIS via the Qt methods and do things like
> calculating intersections for different objects across multiple
> threads at once without having to worry about the GIL at all...
>
> Nyall
>
>
> >
> > - Using Python threads works well when you do everything in Cython (for example) and release the GIL
> >
> > Cheers,
> > Pedro
> >
> >
> > ---- On Mon, 29 Mar 2021 05:00:01 +1000 <qgis-developer-request at lists.osgeo.org> wrote ----
> > > Send QGIS-Developer mailing list submissions to
> > > qgis-developer at lists.osgeo.org
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > > https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > > or, via email, send a message with subject or body 'help' to
> > > qgis-developer-request at lists.osgeo.org
> > >
> > > You can reach the person managing the list at
> > > qgis-developer-owner at lists.osgeo.org
> > >
> > > When replying, please edit your Subject line so it is more specific
> > > than "Re: Contents of QGIS-Developer digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > > 1. Re: Multiprocessing QGIS (Florian El Ahdab)
> > >
> > >
> > > ----------------------------------------------------------------------
> > >
> > > Message: 1
> > > Date: Sun, 28 Mar 2021 16:48:12 +0200
> > > From: Florian El Ahdab <felahdab at gmail.com>
> > > To: Joao Folgado <jfolgado94 at gmail.com>
> > > Cc: David Marteau <dmarteau at 3liz.com>, qgis-developer
> > > <qgis-developer at lists.osgeo.org>
> > > Subject: Re: [QGIS-Developer] Multiprocessing QGIS
> > > Message-ID:
> > > <CAEeudyS4cjKzb9LTRjwyj7fovKgZpNO_hdw66eijgQXbwo61mQ at mail.gmail.com>
> > > Content-Type: text/plain; charset="utf-8"
> > >
> > > Hi.
> > >
> > > I am posting my reply again because I missed the qgis-developer mail list
> > > first...
> > >
> > > It is indeed possible to achieve multiprocessing in QGIS.
> > >
> > > But you should rely on Qt multi thread mechanisms rather than on the python
> > > multiprocessing module.
> > >
> > > Qgis is written in C++, based on the Qt library. Using the Qt
> > > possibilities, even if it is thru python, is less likely to trigger
> > > problems...
> > >
> > > QThread is your friend here.
> > > You can also use classes that are designed in QGis to ease things up, like
> > > QgsTask (https://nocache.qgis.org/api/3.4/classQgsTask.html).
> > >
> > > Regards
> > > Florian.
> > >
> > > Le sam. 27 mars 2021 ? 13:48, Joao Folgado <jfolgado94 at gmail.com> a ?crit :
> > >
> > > > Hi guys,
> > > >
> > > > First of all thank you but, David Marteua and Sebastian Ernst, for the
> > > > answer.
> > > > So is impossible to use multiprocessing in Qgis plugin?
> > > > This is a enormous limitations because Qgis have a lot of data that
> > > > needs to be evaluated in same cases. Nowadays the cpu have 5,7, 9 cores
> > > > and we can not use them in a plugin for example is bad. If some one have
> > > > any idea who we can use multiprocessing in a plugin i will be thankful.
> > > >
> > > > Best regards,
> > > > Jo?o Folgado
> > > >
> > > > David Marteau <dmarteau at 3liz.com> escreveu em sex., 26/03/2021 ?s 08:57 :
> > > >
> > > >> > I am not entirely sure what QGIS' (intended) behavior is in a case like
> > > >> > this. From the top of my head, having read the relevant portion of its
> > > >> > source code, I'd expect at least a second application window likely
> > > >> > followed by non-deterministic behavior (again, depending on the
> > > >> > use-case, likely resulting in a crash). Can someone elaborate?
> > > >>
> > > >> I can confirm that the result is non-deterministic even with headless
> > > >> PyQgis programs.
> > > >>
> > > >> For programs relying heavily on multiprocessing the best strategy is to
> > > >> use a `fork-server` process
> > > >> started at the very beginning (before initializing Qgis) managing the
> > > >> pool of child processes, so that each forked
> > > >> process starts in the same state.
> > > >>
> > > >> David,
> > > >>
> > > >> Le 23/03/2021 ? 19:22, Sebastian M. Ernst a ?crit :
> > > >> > Hi Jo?o, all,
> > > >> >
> > > >> > really interesting question.
> > > >> >
> > > >> >> however when i run the plugin a new instance of application is open
> > > >> >> and give me an error.
> > > >> > I assume that you are using `multiprocessing` [1] from Python's standard
> > > >> > library. Its default behavior is to use process-based parallelism (as
> > > >> > the title in the documentation says). I am assuming that you have not
> > > >> > altered its default behavior (e.g. by configuring it to use threads
> > > >> > instead). This is why you get at least another "worker" process if you
> > > >> > e.g. create a process pool. I have successfully used this method with
> > > >> > QGIS before, but always on Linux and OS X. I am just guessing that you
> > > >> > are doing this on Windows, right?
> > > >> >
> > > >> > Depending on the operating system, `multiprocessing` uses different
> > > >> > methods to create worker processes. On Linux and OS X, it simply forks
> > > >> > [2] the main process. This way, you do not see a second application
> > > >> > window popping up. You "simply" get a second, third, fourth, ... process
> > > >> > "in the background". On Windows, the `fork` syscall does not exist,
> > > >> > neither does something similar [3]. `multiprocessing` therefore starts a
> > > >> > complete new process from scratch. If it was pure Python, it would
> > > >> > simply start a new Python interpreter process. Because it is QGIS, it
> > > >> > opens another instance of QGIS.
> > > >> >
> > > >> > I am not entirely sure what QGIS' (intended) behavior is in a case like
> > > >> > this. From the top of my head, having read the relevant portion of its
> > > >> > source code, I'd expect at least a second application window likely
> > > >> > followed by non-deterministic behavior (again, depending on the
> > > >> > use-case, likely resulting in a crash). Can someone elaborate?
> > > >> >
> > > >> > Best regards,
> > > >> > Sebastian
> > > >> >
> > > >> >
> > > >> > 1: https://docs.python.org/3/library/multiprocessing.html
> > > >> > 2: https://en.wikipedia.org/wiki/Fork_(system_call)
> > > >> > 3: https://stackoverflow.com/q/985281/1672565
> > > >> >
> > > >> >
> > > >> > Am 23.03.21 um 13:08 schrieb Joao Folgado:
> > > >> >> Hi everyone,
> > > >> >>
> > > >> >> I had created a plugin for Qgis and I trying to optimize it. So i use
> > > >> >> in my code the library python that have Queue whit multiprocessing
> > > >> >> however when i run the plugin a new instance of application is open and
> > > >> >> give me an error.
> > > >> >> I had early use this type of multiprocessing before in simple projects
> > > >> >> in university and it works ok.
> > > >> >> Can someone help ? I researched in many forums, i see people whit the
> > > >> >> same question but not with the answer.
> > > >> >> Thank you very much.
> > > >> >>
> > > >> >> Best regards,
> > > >> >> Jo?o Folgado
> > > >> >> Portugal
> > > >> >> --
> > > >> >> JO?O FOLGADO
> > > >> >>
> > > >> >> _______________________________________________
> > > >> >> QGIS-Developer mailing list
> > > >> >> QGIS-Developer at lists.osgeo.org
> > > >> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > > >> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > > >> >>
> > > >> > _______________________________________________
> > > >> > QGIS-Developer mailing list
> > > >> > QGIS-Developer at lists.osgeo.org
> > > >> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > > >> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > > >> _______________________________________________
> > > >> QGIS-Developer mailing list
> > > >> QGIS-Developer at lists.osgeo.org
> > > >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > > >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > > >>
> > > > --
> > > > JO?O FOLGADO
> > > > _______________________________________________
> > > > QGIS-Developer mailing list
> > > > QGIS-Developer at lists.osgeo.org
> > > > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > > > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > > >
> > > -------------- next part --------------
> > > An HTML attachment was scrubbed...
> > > URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210328/84f82603/attachment-0001.html>
> > >
> > > ------------------------------
> > >
> > > Subject: Digest Footer
> > >
> > > _______________________________________________
> > > QGIS-Developer mailing list
> > > QGIS-Developer at lists.osgeo.org
> > > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > >
> > >
> > > ------------------------------
> > >
> > > End of QGIS-Developer Digest, Vol 185, Issue 60
> > > ***********************************************
> > >
> > _______________________________________________
> > QGIS-Developer mailing list
> > QGIS-Developer at lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
More information about the QGIS-Developer
mailing list