<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;mso-fareast-language:EN-US">Thanks Even. From a bit of testing, I can see the time scaling linearly with the requested number of threads despite inputting only 9 pixels, so I suspect that feature
is newer than 3.0.4. I’ll have a look in the changelogs and keep it in mind.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:black">Dr Daniel Evans<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#F6A124">Software Developer<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> gdal-dev <gdal-dev-bounces@lists.osgeo.org>
<b>On Behalf Of </b>Even Rouault<br>
<b>Sent:</b> 06 October 2020 15:12<br>
<b>To:</b> gdal-dev@lists.osgeo.org<br>
<b>Subject:</b> Re: [gdal-dev] GDAL Warp - NUM_THREADS=ALL_CPUS detrimental?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New"">On mardi 6 octobre 2020 13:54:30 CEST Daniel Evans wrote:<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> From reading the documentation more closely, I notice that NUM_THREADS is a<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> separate bit of functionality from -multithread (which only refers to IO),<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> so any thoughts of load balancing weren't correct.<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">>
<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> I suspect GDAL is simply doing exactly as instructed - spawning X threads,<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> even if it won't find work for them. It's therefore up to the user (me!) to<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> measure and work out where the best performance comes from - and blindly<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> throwing in NUM_THREADS=ALL_CPUS is not always a good tactic.<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">The warper has some logic to avoid spawning threads when it thinks this will not be beneficial, but this logic is probably
too simplistic. Basically for a given number of output pixels to warp, it decides to limit the number of threads to number_pixels / 65536 , 65536 being somehow arbitrary (you can actually tune that value with the WARP_THREAD_CHUNK_SIZE config option, but this
is an internal detail for testing only. might be renamed, disappear etc). That is it considers there's no use to spawn a thread if there are less than 65536 pixels to warp. This is probably too small. Another factor that might come into equation is the initial
time to figure out the PROJ coordinate transformation pipeline, but the time it might take is quite unpredictable<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">I'm speaking here about current master behaviour. Didn't check how this was in 3.0.4. This has changed a bit "recently"<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">Even<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">--
<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">Spatialys - Geospatial professional services<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New""><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></span></p>
</div>
<p><a href="https://www.jbagroup.co.uk/sites/www.jbagroup.co.uk/files/documents/15-030%20JBA%20Business%20Continuity%20Briefing%20-%20Latest.pdf"><img width="449" src="https://www.jbagroup.co.uk/imgstore/covid19footer.png" alt="JBA COVID-19 statement"></a></p>
<p><strong><span style="color: #f6a125; font-family:arial;">T</span></strong><span style="text-decoration: none; color: #000000; font-family:arial;"> +44 (0) 1756 799919</span><br>
<a style="text-decoration: none; color: #f6a125; font-family:arial;" href="http://www.jbarisk.com">www.jbarisk.com</a></p>
<p><a href="http://www.jbarisk.com"><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-JBA.png" alt="Visit our website" width="33" height="26"></a> <a href=""><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png" alt="" height="26"></a>
<a href="https://twitter.com/jbarisk"><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-TWITTER.png" alt="Follow us on Twitter" width="33" height="26"></a></p>
<o:p></o:p>
<p></p>
<p><b><span style="background: white; color: black; font-family: "Arial",sans-serif; font-size: 10pt;">Find out more about us here:
<a href="http://www.jbarisk.com/"><font color="#0563c1">www.jbarisk.com</font></a> and
</span></b><b><span style="background: white; color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 10pt;"><a href="http://twitter.com/JBARisk" target="_blank"><font color="#0563c1">follow us on Twitter @JBARisk</font></a> and
<a href="https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT">
<font color="#0563c1">LinkedIn</font></a> </span></b></p>
<p><span style="background: white; color: black; font-family: "Arial",sans-serif; font-size: 8pt;">The JBA Group supports the JBA Trust.</span></p>
<p style="margin: 0cm 0cm 0pt;"><span style="background: white; color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 8pt;">All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named.
If you are not the named addressee you should not disseminate, distribute or copy this e-mail.</span><br>
<span style="color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 8pt;"><span style="background: white;">Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.</span></span><br>
</p>
<p style="margin: 0cm 0cm 0pt;"><span style="color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 8pt;"><span style="background: white;">JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane
North, Broughton, Skipton, North Yorkshire, BD23 3FD, </span></span><span style="background: white; color: black; font-family: "Arial",sans-serif; font-size: 8pt;">Telephone: +441756799919</span></p>
<p> </p>
</body>
</html>