[gdal-dev] gdaladdo taking very long time to build overviews for large extent mosaic.

[Pradeep kumar]

Dear gdal-dev team,

we have to build a mosaic for 350 aerial images. These images are gray
images and have an alpha band. The avg size of  each images is 24000x24000.
The size of final mosaic would be  1676843x1441203. The coordinate system
of input images and final mosaic is EPSG:25832

First approach: virtual mosaic, gdal_translate, gdaladdo
first built a virtual mosaic for all input images with the command
```gdalbuildvrt virtualmosaic.vrt ./*.tif```
This process was fast

``` gdal_translate virtualmosaic.vrt mosaic.tif -ot Byte -of GTiff -co
TILED=YES -co COMPRESS=JPEG  -co BIGTIFF=YES -a_srs EPSG:25832```
This process took almost one and a half days

```gdaladdo -r average --config COMPRESS_OVERVIEW JPEG --config
PHOTOMETRIC_OVERVIEW MINISBLACK --config INTERLEAVE_OVERVIEW PIXEL --config
GDAL_TIFF_OVER_BLOCKSIZE 256 --config BIGTIFF_OVERVIEW YES mosaic.tif 2 4 8
16 32 64 128 256 512 1024 2048 4096 8192```
This was taking way too much time. After three weeks of processing, it did
not even showed any progress on terminal. The mosaic image size is
increasing very slowly (it increased 40 kb in three weeks). We checked
htop. It was using less than 2 percent of cpu processing. Most of the time
the process status id D in red color.
[image: image.png]

This process was taking too much time so we tried the second approach.

Second approach: The idea is to build a mosaic with gdalwarp and build the
overviews with gdaladdo
mosaic command
```gdalwarp -of GTiff -co COMPRESS=JPEG -co TILED=YES -co BIGTIFF=YES
images_path/*.tif image_path/mosaic.tif```
This process was too slow. It was taking around 30 hours to merge one
image. So we thought it would take a long time merge 350 images and build
overviews so we stopped there.

Third approach: We divided all images into 8 sets. First we mosaiced
neighborhood sets using gdalwarp. This process was fast. We followed same
way for next 4 sets. Finally we mosaiced all images with gdalwarp in a few
days. Then we started building overviews with gdaladdo. Again building
overviews was very slow.

We tried with -multi option, increasing the cache memory --config
GDAL_CACHEMAX xxx and -WM xxx but nothing helped us.

Our first and second approach worked well when we mosaiced a few images or
the mosaic extent was small but it is taking too much time to create mosaic
of a large extent.

We have a quite powerful processing machine available which has 20
processing cores and 120 GB RAM.

Finally our questions are:

1) Is there any flaw in our approach to build overviews?
2) If our approach is fine then why it is taking too much time to process
and how we can increase the processing speed?
3) How can we run gdal commands using effectively multiple cores?


Best Regards,
Pradeep Gulla
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181106/bd8d8048/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 55801 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181106/bd8d8048/attachment-0001.png>