[gdal-dev] GDAL Warp - NUM_THREADS=ALL_CPUS detrimental?

[Daniel Evans]

Thanks Even. From a bit of testing, I can see the time scaling linearly with the requested number of threads despite inputting only 9 pixels, so I suspect that feature is newer than 3.0.4. I'll have a look in the changelogs and keep it in mind.

Dr Daniel Evans
Software Developer

From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> On Behalf Of Even Rouault
Sent: 06 October 2020 15:12
To: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] GDAL Warp - NUM_THREADS=ALL_CPUS detrimental?


On mardi 6 octobre 2020 13:54:30 CEST Daniel Evans wrote:

> From reading the documentation more closely, I notice that NUM_THREADS is a

> separate bit of functionality from -multithread (which only refers to IO),

> so any thoughts of load balancing weren't correct.

>

> I suspect GDAL is simply doing exactly as instructed - spawning X threads,

> even if it won't find work for them. It's therefore up to the user (me!) to

> measure and work out where the best performance comes from - and blindly

> throwing in NUM_THREADS=ALL_CPUS is not always a good tactic.



The warper has some logic to avoid spawning threads when it thinks this will not be beneficial, but this logic is probably too simplistic. Basically for a given number of output pixels to warp, it decides to limit the number of threads to number_pixels / 65536 , 65536 being somehow arbitrary (you can actually tune that value with the WARP_THREAD_CHUNK_SIZE config option, but this is an internal detail for testing only. might be renamed, disappear etc). That is it considers there's no use to spawn a thread if there are less than 65536 pixels to warp. This is probably too small. Another factor that might come into equation is the initial time to figure out the PROJ coordinate transformation pipeline, but the time it might take is quite unpredictable



I'm speaking here about current master behaviour. Didn't check how this was in 3.0.4. This has changed a bit "recently"



Even



--

Spatialys - Geospatial professional services

http://www.spatialys.com

[JBA COVID-19 statement]<https://www.jbagroup.co.uk/sites/www.jbagroup.co.uk/files/documents/15-030%20JBA%20Business%20Continuity%20Briefing%20-%20Latest.pdf>

T +44 (0) 1756 799919
www.jbarisk.com<http://www.jbarisk.com>

[Visit our website]<http://www.jbarisk.com> [http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png]  [Follow us on Twitter] <https://twitter.com/jbarisk>

Find out more about us here: www.jbarisk.com<http://www.jbarisk.com/> and follow us on Twitter @JBARisk<http://twitter.com/JBARisk> and LinkedIn<https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT>

The JBA Group supports the JBA Trust.

All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.


JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, Telephone: +441756799919


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20201006/ba7e2270/attachment-0001.html>