[gdal-dev] gdal_translate and multithreading

[Even Rouault]

Selon Pierre Soille <pierre.soille at jrc.ec.europa.eu>:

>
>
>
>
>
>     Even,
>
>
>
>         thank you for your prompt reply.  To be more specific, the
>         successive steps of my procedure are
>
>
>
>         1] gdalwarp of a vrt file referring to 8 bands (stored as
>         gtiffs) of Landsat8 image from UTM into the equivalent EPSG:4326 vrt
> file:
>
>         gdalwarp -of vrt -r cubic -t_srs EPSG:4326 -tr 0.00025 0.00025
>         -te -77.54 69.69 -70.01 72.07 -srcnodata 0 -dstnodata 0
>         LC80260102014095LGN00_REF.vrt
>         LC80260102014095LGN00_REF_EPSG4326.vrt
>
>
>
>         (this first step is almost instantaneous given that it only
>         produces a vrt file.
>
>
>
>         2] translate the obtained vrt file into a gtiff file using
>         gdal_translate:
>
>         gdal_translate -co INTERLEAVE=BAND -co COMPRESS=LZW -co
>         PREDICTOR=2 -co BIGTIFF=IF_SAFER 
>         LC80260102014095LGN00_REF_EPSG4326.vrt
> LC80260102014095LGN00_REF_EPSG4326.tif
>
>
>
>         It is in this second step that htop reports the use of
>         multithreads by gdal_translate.
>
>
>
>         Any idea where I am doing something wrong or why htop reports the use
> of multithreads by gdal_translate in
>         this case?

Unless I'm missing something, there should be no multithreading in your above
use case. I'm not familiar with htop, so I cannot comment. A way of checking if
threads are created is to run the command line with "strace -f" and check if
there are child processes created.

>
>
>
>           As a side note, I noticed that applying the same procedure 8
>           times on the individual bands is much faster (~100 times) than
>           applying it once starting from the vrt file referring to the 8
>           individual bands as described above but also fail to
>           understand such a behaviour.
>
>

It is always difficult to know what bottlenecks are without running and
debugging the use case. My guess: there are some optimizations done when dealing
with pixel interleaved data that might be defeated if you output to a
INTERLEAVE=BAND tiff. Try removing -co INTERLEAVE=BAND


>
>           Thank you again for your time and your amazing contributions
>           to gdal.
>
>
>
>           Best regards,
>
>
>
>           Pierre
>
>
>
>
>
>         On 01/14/2015 10:52 AM, Even Rouault wrote:
>
>
>
>       Pierre,
>
> Nor gdal_translate nor the VRT driver uses multithreading. But it is possible
> that the driver that handles the underlying dataset referenced by the VRT
> uses
> multithreading. There are very few of them however. I can only think of :
> - JP2KAK. You can specify JP2KAK_THREADS=1
> - JP2OpenJPEG. You can specify GDAL_NUM_THREADS=1
>
> Even
>
>
>
>         Hello,
>
> while translating a vrt file into a tif file with gdal_translate, I
> noticed thanks to htop utility that gdal_translate is using
> multithreading (typicallay 4 threads). I fail to find any multithreading
> option for gdal_translate (contrary to gdal_warp that has a multi option
> and where the number of threads can be set with -wo "NUM_THREADS=x").
> Is there any possibility to prevent gdal_translate from multithreading?
> Setting the global environment variable OMP_NUM_THREADS to 1 does not
> help.  I had a quick look to gdal_translate.cpp but did not find any
> explicit reference to multithreading.
>
> This behaviour of gdal_translate is an issue in case of distributed
> computing.  Indeed, cluster nodes with many cores are spending most of
> their time with kernel time to serve a number of threads largely
> exceeding the number of CPUs.
>
> Any hint on how to solve this problem will be greatly appreciated.
>
> Cheers,
>
> Pierre
>
> ------------------------
> Pierre Soille
> EC Joint Research Centre
> Ispra, Italy
> ------------------------
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
>
>
>
>
>
>
>
>
>
>


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com