[QGIS-Developer] Multiprocessing the ArcGIS way?

[Tobias Wendorff]

Hi there,

I know that QGIS has some approaches to support multiple CPUs, but most
of the processing tools and core functions are still running on the main
thread. For example, when starting multiple export jobs (one to GPKG,
one to FGB, one to PostgreSQL), each are sharing a single core.

That's nice when you have a modern CPU, like Zen3, which is optimized to
support this*, but if you're on a Xeon based server or workstation,
you're oftened doomed with slow 2.83 GHz cores.

I'm using a lame workaround from this, which also works for all other
non-parallel tools, like PHP, GDAL etc. I'm just starting my project in
a parallel QGIS and start the export job from there - and zoom: three
cores are used.

That's the same approach, ArcGIS does with their background process.
ArcGIS was single CPU for a long time. I think it was too hard to make
the core application work in parallel. The hack was to handle the
processing over to a background process. This background process was
also able to run in 64-bit, while the main app still was 32-bit.

Can't we do this in QGIS, too? A "background QGIS" without a GUI, just
doing the processing. Actually, this might be pretty similar to QGIS
server? Sure, there might be some problems like accessing temporary
layers, but can't memory addresses of these layers be shared to the
backgroup worker?

Best,
Tobias

*Technical background:
Intel uses large monolithic cores, which were able to provide high
single core performance with low latency in the past. But the problem is
that you get much less of it on a wafer and if something is exposed
wrong, you have high losses. AMD builds rather small cores with its own
caches. So many of them it on the wafer. The cores are connected in a
network. The only problem is that AMD single-core applications are
constantly moving back and forth between the cores to spread the wear
and heat. But of course, the cache and addition data has to be moved
over the network, too. In Zen3, AMD has optimized this communication.