[gdal-dev] GDAL Warp - NUM_THREADS=ALL_CPUS detrimental?

[Daniel Evans]

Hello,

I've found today that when calling gdal.Warp (from Python, though I doubt it matters) with multithreading enabled, setting the option NUM_THREADS=ALL_CPUS results in increased runtime. I find there to be a minimum runtime of ~0.3-0.4s with NUM_THREADS set like this, compared to a runtime of order 0.01-0.03s (measurements aren't particularly accurate) for some tiny test datasets, which is much closer to my expectation.

Is this behaviour expected? It may well be that GDAL has some internal logic to optimise the number of threads, and setting NUM_THREADS=ALL_CPUS is detrimentally overriding that.

I have often seen code examples or guides that suggest setting NUM_THREADS whenever enabling multithreading in GDAL calls. Is this actually poor advice, with GDAL already using the maximum number of available CPUs when it is suitable to do so?

(Runtimes measured in GDAL 3.0.4)

Dr Daniel Evans
Software Developer


[JBA COVID-19 statement]<https://www.jbagroup.co.uk/sites/www.jbagroup.co.uk/files/documents/15-030%20JBA%20Business%20Continuity%20Briefing%20-%20Latest.pdf>

T +44 (0) 1756 799919
www.jbarisk.com<http://www.jbarisk.com>

[Visit our website]<http://www.jbarisk.com> [http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png]  [Follow us on Twitter] <https://twitter.com/jbarisk>

Find out more about us here: www.jbarisk.com<http://www.jbarisk.com/> and follow us on Twitter @JBARisk<http://twitter.com/JBARisk> and LinkedIn<https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT>

The JBA Group supports the JBA Trust.

All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.


JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, Telephone: +441756799919


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20201006/46355c60/attachment.html>