[gdal-dev] Problem when creating overviews for large input files with gdaladdo

[Armin Burger]

Dear all

I ran into a problem when trying to create overviews for large 
LZW-compressed GeoTIFF images using gdaladdo. The TIFF's are internally 
tiled, the file type is Byte. GDAL versions tested are 2.1.1 and 1.11, 
running on 64bit Linux.

The first try was using an input file of size 1038336 x 565248 pixels. 
Setting CPL_DEBUG=ON the processing goes up to the message

...10...20...30...40...50...60...70..GDAL: Potential thrashing on band 1 
of tileindex_nodsm_otsu_2

and then it remains at this "70.." % forever, with no further log 
output. This happened for all files of similar size, on multiple 
machines, on both GDAL versions. Always hangs at this "70.." state, the 
process remains alive but nothing is written any more to the output 
file. I used the command like

gdaladdo -ro --config COMPRESS_OVERVIEW LZW  INPUT_FILE 2 4 8 16 32 64

Trying to specify BIGTIFF or not for the overview did not make any 
difference.

Using a smaller file or converting the large input file to 50% size and 
running the gdaladdo on this smaller file finished successfully, but 
also here I got some warnings of a similar kind, and the process got 
stuck for a while after the ".70..GDAL: Potential thrashing..." warning, 
but then continued to the end. The full log is attached.

An additional observation I made was that the overview file stopping at 
70% was already bigger than the input file.

I was thinking to create single lower resolution files and merging them 
to an overview file, but there's no built-in functionality for this, so 
did not follow it up further.

Could there be a sort of limitation of maximum pixels x*y, or with the 
LZW compression? Any ideas how this could be solved?

Thanks in advance for any hint
Armin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gdaladdo.log
Type: text/x-log
Size: 5367 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20161219/a00e777d/attachment.bin>