[gdal-dev] TIFFMergeFieldInfo error when adding overviews to large tiff

[Eli Adam]

Hi all,

I'm adding internal overviews to a ~20G tiff file and I get this error
(1000+ times):
gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config
COMPRESS_OVERVIEW PACKBITS
ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields
array (1410046 elements of 16 bytes each)
gdaladdo still completes and the file seems to work fine.  What does
that error mean?  Does it impact the output file or overviews?  Is
there a different way I should approach this?  Are there any general
rules for which compression to select?  I've found JPEG-In-TIFF, using
PHOTOMETRIC=YCBCR to work very well for RGB, but otherwise I'm unsure
of the best compression for the type of raster (in this case single
band, byte, colormap).

I just checked my version and I am running 1.9.0 on a older modest XP
laptop (I figured that I would have been running something more
recent).  I can try again with a nightly or compile on Ubuntu or use a
computer with more resources (the process takes a long time to run, so
reporting back on those will take some time).

Here is information (I've omitted some previous mosaicing and
reprojecting) of how I got to this point and some gdalinfo reports:
gdal_translate mosaic_4326.vrt topo_mosaic_4326.tif -co
COMPRESS=PACKBITS -co TILED=YES -co BIGTIFF=YES --config GDAL_CACHEMAX
400

gdaladdo topo_mosaic_4326.tif 2 4 8 16 32 64 128 256 512 1024 --config
COMPRESS_OVERVIEW PACKBITS
ERROR 1: topo_mosaic_4326.tif:Failed to allocate memory for for fields
array (1410046 elements of 16 bytes each)

original input files:
>gdalinfo ID_Cuddy_Mountain_20110715_TM_geo.tif
Driver: GTiff/GeoTIFF
Files: ID_Cuddy_Mountain_20110715_TM_geo.tif
Size is 4880, 6845
Coordinate System is:
PROJCS["NAD83 / UTM zone 11N",
    GEOGCS["NAD83",
        DATUM["North_American_Datum_1983",
            SPHEROID["GRS 1980",6378137,298.2572221010002,
                AUTHORITY["EPSG","7019"]],
            AUTHORITY["EPSG","6269"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4269"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",-117],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","26911"]]
Origin = (509873.309826691810000,4969095.033418485900000)
Pixel Size = (2.032010116947724,-2.032010116947745)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  509873.310, 4969095.033) (116d52'30.00"W, 44d52'30.74"N)
Lower Left  (  509873.310, 4955185.924) (116d52'30.97"W, 44d44'59.97"N)
Upper Right (  519789.519, 4969095.033) (116d44'58.04"W, 44d52'30.00"N)
Lower Right (  519789.519, 4955185.924) (116d44'59.99"W, 44d44'59.23"N)
Center      (  514831.415, 4962140.479) (116d48'44.75"W, 44d48'45.05"N)
Band 1 Block=4880x1 Type=Byte, ColorInterp=Palette
  NoData Value=195
  Color Table (RGB with 256 entries)
    0: 0,0,0,255
    1: 128,0,0,255
    2: 0,128,0,255
    3: 128,128,0,255
...
  254: 0,255,255,255
  255: 255,255,255,255

final output file with overviews:
>gdalinfo -checksum topo_mosaic_4326.tif
Driver: GTiff/GeoTIFF
Files: topo_mosaic_4326.tif
Size is 398449, 210532
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-124.732439702451200,46.380533997595272)
Pixel Size = (0.000020993062915,-0.000020993062915)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=PACKBITS
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (-124.7324397,  46.3805340) (124d43'56.78"W, 46d22'49.92"N)
Lower Left  (-124.7324397,  41.9608225) (124d43'56.78"W, 41d57'38.96"N)
Upper Right (-116.3677748,  46.3805340) (116d22' 3.99"W, 46d22'49.92"N)
Lower Right (-116.3677748,  41.9608225) (116d22' 3.99"W, 41d57'38.96"N)
Center      (-120.5501072,  44.1706782) (120d33' 0.39"W, 44d10'14.44"N)
Band 1 Block=256x256 Type=Byte, ColorInterp=Palette
  Checksum=45365
  NoData Value=195
  Overviews: 199225x105266, 99613x52633, 49807x26317, 24904x13159, 12452x6580, 6
226x3290, 3113x1645, 1557x823, 779x412, 390x206
  Overviews checksum: 27605, 16665, 58099, 63342, 4876, 64021, 44152, 14528, 574
63, 15006
  Color Table (RGB with 256 entries)
    0: 0,0,0,255
    1: 128,0,0,255
    2: 0,128,0,255
    3: 128,128,0,255
...
  253: 255,0,255,255
  254: 0,255,255,255
  255: 255,255,255,255

Best Regards, Eli