[gdal-dev] BigTIFF Overviews ( OVR files )

[Even Rouault]

2 things :

1)  Christopher Schmidt is right in suggesting to increase  GDAL_CACHEMAX (the 
default is 40 MB). In your use case (building external overviews with 
compression), the analysis of the algorithm (and a bit of experimentation to 
confirm ;-)) shows that that the minimum cache size is :
source_block_width * dst_block_height * overview_factor * sizeof(datatype) =
119999 * 128 * 2 * 2 = 61, 4 MB 
(to be exact, you should also add to that a bit of extra space for the size of 
a target block (128 * 128 * 2) to hold in the cache). So specifying --config 
GDAL_CACHEMAX 65 should be far sufficient to avoid source band thrashing. 
Increasing it more than that wouldn't help.

More generally, to diagnose performance problems, setting CPL_DEBUG=ON can be 
interesting. If you see, a warning like 'GDAL: Potential thrashing on band 1 
of', this is an indication to try increasing GDAL_CACHEMAX

2) --config PREDICTOR 2 will be ignored. This is a creation option of the 
GTiff driver, but there's no corresponding configuration option in the GTiff 
overview building code. This could be easily added if needed.

Le Tuesday 09 February 2010 23:32:55 David Fogel, vous avez écrit :
> The gdaladdo command is below (for a single overview):
>
> D:\SRTM\Central_North> "c:\Tamas\bin\gdal\apps\gdaladdo.exe" -r
> nearest -ro --config BIGTIFF_OVERVIEW YES --config COMPRESS_OVERVIEW
> LZW --config PREDICTOR 2 Central_North.tif 2
>
>
> On Tue, Feb 9, 2010 at 2:26 PM, Even Rouault
>
> <even.rouault at mines-paris.org> wrote:
> > And the exact gdaladdo command line you're using ? (must be sure if it's
> > internal or external overviews)
> >
> > Le Tuesday 09 February 2010 23:22:05 David Fogel, vous avez écrit :
> >> Hi Even:
> >>
> >> The GeoTIFF file is described below (output from gdalinfo).  The
> >> pre-compiled code is from Tamas' site:
> >> http://vbkto.dyndns.org/sdk/
> >> I am using the MSVC2008 version ( scroll to the bottom of the page ).
> >> This ought to be v1.7.0 with some or all of the code for HFA files
> >> folded into it.
> >>
> >> The workstation is running Windows 7 64-bit w/ 8 GB of RAM and too
> >> much disk space to know what to do with .. for the moment.
> >>
> >> Thanks.
> >> = David
> >>
> >> ----- % cut here % -----
> >> Driver: GTiff/GeoTIFF
> >> Files: Central_North.tif
> >> Size is 119999, 74401
> >> Coordinate System is:
> >> GEOGCS["WGS 84",
> >>     DATUM["WGS_1984",
> >>         SPHEROID["WGS 84",6378137,298.257223563,
> >>             AUTHORITY["EPSG","7030"]],
> >>         AUTHORITY["EPSG","6326"]],
> >>     PRIMEM["Greenwich",0],
> >>     UNIT["degree",0.0174532925199433],
> >>     AUTHORITY["EPSG","4326"]]
> >> Origin = (-31.999583000000001,60.000416333276846)
> >> Pixel Size = (0.000833333333333,-0.000833333333333)
> >> Metadata:
> >>   AREA_OR_POINT=Area
> >> Image Structure Metadata:
> >>   COMPRESSION=LZW
> >>   INTERLEAVE=BAND
> >> Corner Coordinates:
> >> Upper Left  ( -31.9995830,  60.0004163) ( 31d59'58.50"W, 60d 0'1.50"N)
> >> Lower Left  ( -31.9995830,  -2.0004170) ( 31d59'58.50"W,  2d 0'1.50"S)
> >> Upper Right (  67.9995837,  60.0004163) ( 67d59'58.50"E, 60d 0'1.50"N)
> >> Lower Right (  67.9995837,  -2.0004170) ( 67d59'58.50"E,  2d 0'1.50"S)
> >> Center      (  18.0000003,  28.9999997) ( 18d 0'0.00"E, 29d 0'0.00"N)
> >> Band 1 Block=119999x1 Type=Int16, ColorInterp=Gray
> >>   NoData Value=-32768
> >>   Metadata:
> >>     LAYER_TYPE=athematic
> >> ----- % cut here % -----
> >>
> >> On Tue, Feb 9, 2010 at 2:09 PM, Even Rouault
> >>
> >> <even.rouault at mines-paris.org> wrote:
> >> > Could you append the output of gdalinfo on the TIF you make overviews
> >> > on and the exact gdaladdo you're using (mainly if you use one
> >> > particular resampling algorithm) ?
> >> >
> >> > Le Tuesday 09 February 2010 22:57:26 David Fogel, vous avez écrit :
> >> >> Hi:
> >> >>
> >> >> In the process of moving data into GeoTIFF, I am re-creating pyramids
> >> >> (OVR files).  The good news: It takes fewer than ten minutes to
> >> >> create an LZW Horizontal Differenced compressed 3.5 GB file on a
> >> >> Windows 7 box using GDAL 1.70+ ( via Tamas Szekeres site; 64-bit
> >> >> precompiled ).
> >> >>
> >> >> The OVR files are problematic.  Using gdal_translate, I can resize by
> >> >> 50% in less than ten minutes.  Using gdaladdo, the first layer ( "2"
> >> >> ) takes hours to complete.
> >> >>
> >> >> This is for SRTM 16-bit data.  It appears that something is amiss.
> >> >>
> >> >> Tips and tricks?
> >> >>
> >> >> Thanks!
> >> >> = David
> >> >> _______________________________________________
> >> >> gdal-dev mailing list
> >> >> gdal-dev at lists.osgeo.org
> >> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev