[gdal-dev] TIFFMergeFieldInfo error when adding overviews to large tiff

[Eli Adam]

Even,

Thanks for checking into this and the ideas.

Eli

On Tue, Apr 16, 2013 at 10:18 AM, Even Rouault
<even.rouault at mines-paris.org> wrote:
> Le lundi 15 avril 2013 23:47:33, Eli Adam a écrit :
>> Hi all,
>>
>> I'm adding internal overviews to a ~20G tiff file and I get this error
>> (1000+ times):
>> gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config
>> COMPRESS_OVERVIEW PACKBITS
>> ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields
>> array (1410046 elements of 16 bytes each)
>> gdaladdo still completes and the file seems to work fine.  What does
>> that error mean?  Does it impact the output file or overviews?  Is
>> there a different way I should approach this?  Are there any general
>> rules for which compression to select?  I've found JPEG-In-TIFF, using
>> PHOTOMETRIC=YCBCR to work very well for RGB, but otherwise I'm unsure
>> of the best compression for the type of raster (in this case single
>> band, byte, colormap).
>
> That's interesting. I've tried to reproduce your setup with an artificial blank
> input file of the same dimensions and here's what I get (after several hours):
>
> $ gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config
> COMPRESS_OVERVIEW PACKBITS
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> Current VSI memory usage        : 0 bytes
> Maximum VSI memory usage        : 2361761086 bytes
> Number of calls to VSIMalloc()  : 43981823
> Number of calls to VSICalloc()  : 1259
> Number of calls to VSIRealloc() : 17378556
> Number of calls to VSIFree()    : 43983082
> VSIMalloc + VSICalloc - VSIFree : 0
>
> The interesting part is the memory usage report. It is based on an
> instrumented version of GDAL that instruments each memory allocation routine.
> This is with the internal version of libtiff that redirects the TIFF memory
> allocation routines to the GDAL ones. So apparently there wouldn't be a memory
> leak (some valgrind'ing should be done on a smaller file to confirm), but a
> maximum memory usage of 2.3 GB ! No wonder why it fails on a Windows 32 bit
> (my test was done a Linux 64bit). I'm not sure why such a high memory use
> occurs. Some investigation should be done.
>
> As far as the error message you've seen, I wouldn't swear that there's not
> some pixel corruption in the result you get...
>
> As a potential track for a workaround (completely untested), perhaps you could
> try to do :
>
> $ gdaladdo topo_mosaic_4326.tif 2 --config COMPRESS_OVERVIEW PACKBITS
> $ gdaladdo topo_mosaic_4326.tif 4 --config COMPRESS_OVERVIEW PACKBITS
> $ gdaladdo topo_mosaic_4326.tif 8 --config COMPRESS_OVERVIEW PACKBITS
> $ gdaladdo topo_mosaic_4326.tif 16 --config COMPRESS_OVERVIEW PACKBITS
> etc..
>
>
>