[gdal-dev] GDAL Warp - NUM_THREADS=ALL_CPUS detrimental?

[Even Rouault]

On mardi 6 octobre 2020 13:54:30 CEST Daniel Evans wrote:
> From reading the documentation more closely, I notice that NUM_THREADS is a
> separate bit of functionality from -multithread (which only refers to IO),
> so any thoughts of load balancing weren't correct.
> 
> I suspect GDAL is simply doing exactly as instructed - spawning X threads,
> even if it won't find work for them. It's therefore up to the user (me!) to
> measure and work out where the best performance comes from - and blindly
> throwing in NUM_THREADS=ALL_CPUS is not always a good tactic.

The warper has some logic to avoid spawning threads when it thinks this will not be 
beneficial, but this logic is probably too simplistic. Basically for a given number of output 
pixels to warp, it decides to limit the number of threads to  number_pixels / 65536 , 65536 
being somehow arbitrary (you can actually tune that value with the 
WARP_THREAD_CHUNK_SIZE config option, but this is an internal detail for testing only. 
might be renamed, disappear etc). That is it considers there's no use to spawn a thread if 
there are less than 65536 pixels to warp. This is probably too small. Another factor that might 
come into equation is the initial time to figure out the PROJ coordinate transformation 
pipeline, but the time it might take is quite unpredictable

I'm speaking here about current master behaviour. Didn't check how this was in 3.0.4. This has 
changed a bit "recently"

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20201006/220c1095/attachment.html>