[QGIS-Developer] Multiprocessing the ArcGIS way?

[Nyall Dawson]

On Mon, 19 Oct 2020 at 12:22, Tobias Wendorff
<tobias.wendorff at tu-dortmund.de> wrote:
>
> Hi there,
>
> I know that QGIS has some approaches to support multiple CPUs, but most
> of the processing tools and core functions are still running on the main
> thread. For example, when starting multiple export jobs (one to GPKG,
> one to FGB, one to PostgreSQL), each are sharing a single core.
>
> That's nice when you have a modern CPU, like Zen3, which is optimized to
> support this*, but if you're on a Xeon based server or workstation,
> you're oftened doomed with slow 2.83 GHz cores.
>
> I'm using a lame workaround from this, which also works for all other
> non-parallel tools, like PHP, GDAL etc. I'm just starting my project in
> a parallel QGIS and start the export job from there - and zoom: three
> cores are used.
>
> That's the same approach, ArcGIS does with their background process.
> ArcGIS was single CPU for a long time. I think it was too hard to make
> the core application work in parallel. The hack was to handle the
> processing over to a background process. This background process was
> also able to run in 64-bit, while the main app still was 32-bit.
>
> Can't we do this in QGIS, too?

We already do! Definitely for exporting vectors to other vector
formats everything is run in a background thread.

I'm curious to hear which version you're using, and what your exact workflow is.

Nyall

A "background QGIS" without a GUI, just
> doing the processing. Actually, this might be pretty similar to QGIS
> server? Sure, there might be some problems like accessing temporary
> layers, but can't memory addresses of these layers be shared to the
> backgroup worker?
>
> Best,
> Tobias
>
> *Technical background:
> Intel uses large monolithic cores, which were able to provide high
> single core performance with low latency in the past. But the problem is
> that you get much less of it on a wafer and if something is exposed
> wrong, you have high losses. AMD builds rather small cores with its own
> caches. So many of them it on the wafer. The cores are connected in a
> network. The only problem is that AMD single-core applications are
> constantly moving back and forth between the cores to spread the wear
> and heat. But of course, the cache and addition data has to be moved
> over the network, too. In Zen3, AMD has optimized this communication.
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer