<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">From reading the documentation more closely, I notice that NUM_THREADS is a separate bit of functionality from –multithread (which only refers to IO), so any thoughts of load balancing weren’t
correct.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">I suspect GDAL is simply doing exactly as instructed – spawning X threads, even if it won’t find work for them. It’s therefore up to the user (me!) to measure and work out where the best performance
comes from – and blindly throwing in NUM_THREADS=ALL_CPUS is not always a good tactic.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:black;mso-fareast-language:EN-GB">Dr Daniel Evans<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#F6A124;mso-fareast-language:EN-GB">Software Developer<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:EN-GB">From:</span></b><span lang="EN-US" style="mso-fareast-language:EN-GB"> Daniel Evans
<br>
<b>Sent:</b> 06 October 2020 14:38<br>
<b>To:</b> gdal-dev@lists.osgeo.org<br>
<b>Subject:</b> GDAL Warp - NUM_THREADS=ALL_CPUS detrimental?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">I’ve found today that when calling gdal.Warp (from Python, though I doubt it matters) with multithreading enabled, setting the option NUM_THREADS=ALL_CPUS results in increased runtime. I find
there to be a minimum runtime of ~0.3-0.4s with NUM_THREADS set like this, compared to a runtime of order 0.01-0.03s (measurements aren’t particularly accurate) for some tiny test datasets, which is much closer to my expectation.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">Is this behaviour expected? It may well be that GDAL has some internal logic to optimise the number of threads, and setting NUM_THREADS=ALL_CPUS is detrimentally overriding that.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">I have often seen code examples or guides that suggest setting NUM_THREADS whenever enabling multithreading in GDAL calls. Is this actually poor advice, with GDAL already using the maximum number
of available CPUs when it is suitable to do so?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif">(Runtimes measured in GDAL 3.0.4)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:black;mso-fareast-language:EN-GB">Dr Daniel Evans<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#F6A124;mso-fareast-language:EN-GB">Software Developer<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p><a href="https://www.jbagroup.co.uk/sites/www.jbagroup.co.uk/files/documents/15-030%20JBA%20Business%20Continuity%20Briefing%20-%20Latest.pdf"><img width="449" src="https://www.jbagroup.co.uk/imgstore/covid19footer.png" alt="JBA COVID-19 statement"></a></p>
<p><strong><span style="color: #f6a125; font-family:arial;">T</span></strong><span style="text-decoration: none; color: #000000; font-family:arial;"> +44 (0) 1756 799919</span><br>
<a style="text-decoration: none; color: #f6a125; font-family:arial;" href="http://www.jbarisk.com">www.jbarisk.com</a></p>
<p><a href="http://www.jbarisk.com"><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-JBA.png" alt="Visit our website" width="33" height="26"></a> <a href=""><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png" alt="" height="26"></a>
<a href="https://twitter.com/jbarisk"><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-TWITTER.png" alt="Follow us on Twitter" width="33" height="26"></a></p>
<o:p></o:p>
<p></p>
<p><b><span style="background: white; color: black; font-family: "Arial",sans-serif; font-size: 10pt;">Find out more about us here:
<a href="http://www.jbarisk.com/"><font color="#0563c1">www.jbarisk.com</font></a> and
</span></b><b><span style="background: white; color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 10pt;"><a href="http://twitter.com/JBARisk" target="_blank"><font color="#0563c1">follow us on Twitter @JBARisk</font></a> and
<a href="https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT">
<font color="#0563c1">LinkedIn</font></a> </span></b></p>
<p><span style="background: white; color: black; font-family: "Arial",sans-serif; font-size: 8pt;">The JBA Group supports the JBA Trust.</span></p>
<p style="margin: 0cm 0cm 0pt;"><span style="background: white; color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 8pt;">All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named.
If you are not the named addressee you should not disseminate, distribute or copy this e-mail.</span><br>
<span style="color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 8pt;"><span style="background: white;">Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.</span></span><br>
</p>
<p style="margin: 0cm 0cm 0pt;"><span style="color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 8pt;"><span style="background: white;">JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane
North, Broughton, Skipton, North Yorkshire, BD23 3FD, </span></span><span style="background: white; color: black; font-family: "Arial",sans-serif; font-size: 8pt;">Telephone: +441756799919</span></p>
<p> </p>
</body>
</html>