[gdal-dev] gdaladdo taking very long time to build overviews for large extent mosaic.

[Even Rouault]

On mardi 6 novembre 2018 11:08:14 CET Pradeep kumar wrote:
> Dear gdal-dev team,
> 
> we have to build a mosaic for 350 aerial images. These images are gray
> images and have an alpha band. The avg size of  each images is 24000x24000.
> The size of final mosaic would be  1676843x1441203. The coordinate system
> of input images and final mosaic is EPSG:25832
> 
> First approach: virtual mosaic, gdal_translate, gdaladdo
> first built a virtual mosaic for all input images with the command
> ```gdalbuildvrt virtualmosaic.vrt ./*.tif```
> This process was fast
> 
> ``` gdal_translate virtualmosaic.vrt mosaic.tif -ot Byte -of GTiff -co
> TILED=YES -co COMPRESS=JPEG  -co BIGTIFF=YES -a_srs EPSG:25832```
> This process took almost one and a half days
> 
> ```gdaladdo -r average --config COMPRESS_OVERVIEW JPEG --config
> PHOTOMETRIC_OVERVIEW MINISBLACK --config INTERLEAVE_OVERVIEW PIXEL --config
> GDAL_TIFF_OVER_BLOCKSIZE 256 --config BIGTIFF_OVERVIEW YES mosaic.tif 2 4 8
> 16 32 64 128 256 512 1024 2048 4096 8192```
> This was taking way too much time. After three weeks of processing, it did
> not even showed any progress on terminal. The mosaic image size is
> increasing very slowly (it increased 40 kb in three weeks). We checked
> htop. It was using less than 2 percent of cpu processing. Most of the time
> the process status id D in red color.

Yes, this is a well known issue due to costly 'context switching' in the 
libtiff library. A solution is to create external overviews, and one level at 
a time, and repeat the creation of a new level on the .ovr of the previous 
level.

So:
gdaladdo -ro [options here] mosaic.tif 2 (actually you can do it on the .vrt 
file directly to, and rename the .vrt.ovr as mosaic.tif.ovr)
gdaladdo -ro [options here] mosaic.tif.ovr 2
gdaladdo -ro [options here] mosaic.tif.ovr.ovr 2
gdaladdo -ro [options here] mosaic.tif.ovr.ovr.ovr 2
etc

If you want a single file at the end, you then do:

gdal_translate mosaic.tif final.tif -co COPY_SRC_OVERVIEWS=YES [other options]

In that case, you'd better drop the JPEG compression in the first 
gdal_translate and gdaladdo steps, and only apply it in the final stage.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com