[gdal-dev] TIFFMergeFieldInfo error when adding overviews to large tiff

[Even Rouault]

Le lundi 15 avril 2013 23:47:33, Eli Adam a écrit :
> Hi all,
> 
> I'm adding internal overviews to a ~20G tiff file and I get this error
> (1000+ times):
> gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config
> COMPRESS_OVERVIEW PACKBITS
> ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields
> array (1410046 elements of 16 bytes each)
> gdaladdo still completes and the file seems to work fine.  What does
> that error mean?  Does it impact the output file or overviews?  Is
> there a different way I should approach this?  Are there any general
> rules for which compression to select?  I've found JPEG-In-TIFF, using
> PHOTOMETRIC=YCBCR to work very well for RGB, but otherwise I'm unsure
> of the best compression for the type of raster (in this case single
> band, byte, colormap).

That's interesting. I've tried to reproduce your setup with an artificial blank 
input file of the same dimensions and here's what I get (after several hours):

$ gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config 
COMPRESS_OVERVIEW PACKBITS
0...10...20...30...40...50...60...70...80...90...100 - done.
Current VSI memory usage        : 0 bytes
Maximum VSI memory usage        : 2361761086 bytes
Number of calls to VSIMalloc()  : 43981823
Number of calls to VSICalloc()  : 1259
Number of calls to VSIRealloc() : 17378556
Number of calls to VSIFree()    : 43983082
VSIMalloc + VSICalloc - VSIFree : 0

The interesting part is the memory usage report. It is based on an 
instrumented version of GDAL that instruments each memory allocation routine. 
This is with the internal version of libtiff that redirects the TIFF memory 
allocation routines to the GDAL ones. So apparently there wouldn't be a memory 
leak (some valgrind'ing should be done on a smaller file to confirm), but a 
maximum memory usage of 2.3 GB ! No wonder why it fails on a Windows 32 bit 
(my test was done a Linux 64bit). I'm not sure why such a high memory use 
occurs. Some investigation should be done.

As far as the error message you've seen, I wouldn't swear that there's not 
some pixel corruption in the result you get...

As a potential track for a workaround (completely untested), perhaps you could 
try to do :

$ gdaladdo topo_mosaic_4326.tif 2 --config COMPRESS_OVERVIEW PACKBITS
$ gdaladdo topo_mosaic_4326.tif 4 --config COMPRESS_OVERVIEW PACKBITS
$ gdaladdo topo_mosaic_4326.tif 8 --config COMPRESS_OVERVIEW PACKBITS
$ gdaladdo topo_mosaic_4326.tif 16 --config COMPRESS_OVERVIEW PACKBITS
etc..