[Qgis-user] excessive threads?

[Mike Stanton]

ALCON,  We are processing data at 4cm resolution from 4000 ft altitude that gives me a DSM Ortho of about 800 GB -1.3 TB depending on the bit depth. My folders have a rile count of over 88,000 Tiles @ 2048 X 2048 with a GSD of ~.076 depending on the processing you are doing I am guessing you can get 8-16 tiles per thread when doing a mosaic operation or even an export. No mention of the tool being used was given, I am doing a lot of reprojects on that file base I described and if I go into multi-thread mode I know it will kick that number up.

Again, I am used to what I see from another tool that uses the GDAL Library for processing and it does get up over 140 threads when it is doing it processing

From: chris hermansen <clhermansen at gmail.com>
Sent: Saturday, July 30, 2022 3:57 PM
To: Mike Stanton <mfstanton at geospan.com>
Cc: Greg Troxel <gdt at lexort.com>; qgis-user <qgis-user at lists.osgeo.org>
Subject: Re: [Qgis-user] excessive threads?

Mike and list,

Mike, would you please explain why the surface area would be related to the number of threads?

1000 square miles or ~ 250.000 hectares in my units is big but not huge. We have a 7 million hectare vegetation inventory we're updating now.

Do you have a bunch of small tiles? Are you thinking that 1 tile = 1 thread?

On Sat, Jul 30, 2022, 12:46 Mike Stanton <mfstanton at geospan.com<mailto:mfstanton at geospan.com>> wrote:
Chris, it really depends on how large your project is. Some of my projects cover 1000 sq miles.  Usually I don’t use QGIS directly for that, I break it down to smaller pieces.

From: Qgis-user <qgis-user-bounces at lists.osgeo.org<mailto:qgis-user-bounces at lists.osgeo.org>> On Behalf Of chris hermansen via Qgis-user
Sent: Saturday, July 30, 2022 2:37 PM
To: Greg Troxel <gdt at lexort.com<mailto:gdt at lexort.com>>
Cc: qgis-user <qgis-user at lists.osgeo.org<mailto:qgis-user at lists.osgeo.org>>
Subject: Re: [Qgis-user] excessive threads?

Greg and list

On Sat, Jul 30, 2022 at 10:38 AM Greg Troxel via Qgis-user <qgis-user at lists.osgeo.org<mailto:qgis-user at lists.osgeo.org>> wrote:

(I realize excessive is relative.  Back when I was young, we didn't have
any threads at all....)

My desktop is NetBSD 9, I ran out of threads, and found that qgis 3.22.8
was using 157 threads, much more than I expected.

[stuff deleted]

I really don't understand:

  What is qgis using threads for?

  Do others see large (100 or so, vs 10) thread counts?

  Why are there 157 (really, why are there more than about 10-20)?
Seems kind of surprising to me as well.  I get that some applications could benefit by parallel execution, and of course separating rendering from file or other I/O seems reasonable, but 157 is a lot of threads for just reading data and visualizing it.

As far as I know, PostgreSQL doesn't even use threads.  See for example this discussion https://dba.stackexchange.com/questions/251935/number-of-worker-threads-available-in-postgresql

About SQLite I have no idea.

Does it seem like you have one thread per layer for reading plus one for rendering plus one for user  input?


--
Chris Hermansen · clhermansen "at" gmail "dot" com

C'est ma façon de parler.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20220730/0bf32a6e/attachment.htm>