[Qgis-user] excessive threads?

[Mike Stanton]

Yes I see a lot of threads. The more tne merrier, the faster the throughput. 

I am running an I9 9980XE with 18 cores.  What having the threads running does is it allows computing to happen while I/O is running in the foreground. To make most efficient use of multiple threads the workload has to be one that can be partitioned in neat little buckets so there is no data conflict. A large Aerial Triangulation or Digital Ortho (Mosaic) operation on image tiles that were physically separated in memory (Column and Row in memory)would be Ideal candidates

One thing I found was that for HPC applications Hyperthreading can actually cost performance and should be turned off.  This has been borne out by benchmarks and can cost as much as 10% of your CPU performance. You are better just relying on the normal Affinized Core in a multi core processor switching architecture if it is available to you (supermicro or Dell PowerEdge platforms) . Hyperthreading is an automatic hardware function switching the active thread on occurrence of a CPU long wait state.  That switch takes time because it requires a state save. Core switching does too but it is less random requiring an execution event (I/O or Hard Page Fault) to trigger it.

-----Original Message-----
From: Qgis-user <qgis-user-bounces at lists.osgeo.org> On Behalf Of Greg Troxel via Qgis-user
Sent: Saturday, July 30, 2022 12:38 PM
To: qgis-user at lists.osgeo.org
Subject: [Qgis-user] excessive threads?


(I realize excessive is relative.  Back when I was young, we didn't have any threads at all....)

My desktop is NetBSD 9, I ran out of threads, and found that qgis 3.22.8 was using 157 threads, much more than I expected.

Yes, I know 157 is not a super large number, and that the limit of 1024 threads is not tuned for hardware that's good enough to really use qgis.
(For various reasons, partially historical inertia, partially because most NetBSD people have a computing history that tends towards what today might be called minimalism, and because NetBSD is used on old hardware and hardware with very little memory (if it doesn't run in 128 MB we think that's a bug), some of the default limits are too low.
Obviously when doing big things those limits need to be increased, but it tends to surface resource usage that's higher than perhaps it should be.  This is aligned with the idea that developers should have to use the programs they are developing on at laest 10 year old normal hardware to see what it's like.)

My qgis instance has a lot of layers, but many are geopackage, many are shapefiles, a couple are postgis, a few are online TMS, a few are TIF, and a few are jp2 tiles.  While it takes 20s to start up (machine is
2014 4-core, 24G RAM, 2T SSD), it's fast once started.

I really don't understand:

  What is qgis using threads for?

  Do others see large (100 or so, vs 10) thread counts?

  Why are there 157 (really, why are there more than about 10-20)?

Thanks,
Greg