[gdal-dev] BigTIFF Overviews ( OVR files )

Tue Feb 9 18:54:00 EST 2010

Even, Christopher:

I have started building a new overview with the GDAL_CACHEMAX set to
65.  Already, I can see the improvement.

It is a bit embarrassing to inquire why GDAL_CACHEMAX did not show
such a change using 2048 or a larger number?  I am seeing the
thrashing that you mentioned @ 65 so I will increase GDAL_CACHEMAX a
little bit at a time.

Re: Horizontal Differencing ( LZW Compressed across 16-bits ), the
results for elevation data are quite noticeable.  The frequency at
which the high and low-order bytes change is very different.  The
uncompressed ASTER GDEM V001 ( elevation and metadata ( NUM ) files )
can be fit on a disk with about 150 GB.  That's much better than
uncompressed ( 1.2 TB ) and LZW 8-bit compression ( 350 GB IIRC ).  As
many people probably know, a 16-bit raster dataset may increase in
size with LZW 8-bit compression.

I have often wondered if the LZW horizontal differencing could be
extended to 32-bits and what the implications would be.  The source
code for the TIFF LZW compression options is fairly clean ( dates back
to the mid-80s I think ).  Could a floating point number be handled
and at what cost ( compressed file size vs. uncompressing for
real-time / online access )?

Merci beaucoup / Thanks,
= David

On Tue, Feb 9, 2010 at 3:24 PM, Even Rouault
<even.rouault at mines-paris.org> wrote:
> 2 things :
>
> 1)  Christopher Schmidt is right in suggesting to increase  GDAL_CACHEMAX (the
> default is 40 MB). In your use case (building external overviews with
> compression), the analysis of the algorithm (and a bit of experimentation to
> confirm ;-)) shows that that the minimum cache size is :
> source_block_width * dst_block_height * overview_factor * sizeof(datatype) =
> 119999 * 128 * 2 * 2 = 61, 4 MB
> (to be exact, you should also add to that a bit of extra space for the size of
> a target block (128 * 128 * 2) to hold in the cache). So specifying --config
> GDAL_CACHEMAX 65 should be far sufficient to avoid source band thrashing.
> Increasing it more than that wouldn't help.
>
> More generally, to diagnose performance problems, setting CPL_DEBUG=ON can be
> interesting. If you see, a warning like 'GDAL: Potential thrashing on band 1
> of', this is an indication to try increasing GDAL_CACHEMAX
>
> 2) --config PREDICTOR 2 will be ignored. This is a creation option of the
> GTiff driver, but there's no corresponding configuration option in the GTiff
> overview building code. This could be easily added if needed.
>
> Le Tuesday 09 February 2010 23:32:55 David Fogel, vous avez écrit :
>> The gdaladdo command is below (for a single overview):
>>
>> D:\SRTM\Central_North> "c:\Tamas\bin\gdal\apps\gdaladdo.exe" -r
>> nearest -ro --config BIGTIFF_OVERVIEW YES --config COMPRESS_OVERVIEW
>> LZW --config PREDICTOR 2 Central_North.tif 2
>>
>>
>> On Tue, Feb 9, 2010 at 2:26 PM, Even Rouault
>>
>> <even.rouault at mines-paris.org> wrote:
>> > And the exact gdaladdo command line you're using ? (must be sure if it's
>> > internal or external overviews)
>> >
>> > Le Tuesday 09 February 2010 23:22:05 David Fogel, vous avez écrit :
>> >> Hi Even:
>> >>
>> >> The GeoTIFF file is described below (output from gdalinfo).  The
>> >> pre-compiled code is from Tamas' site:
>> >> http://vbkto.dyndns.org/sdk/
>> >> I am using the MSVC2008 version ( scroll to the bottom of the page ).
>> >> This ought to be v1.7.0 with some or all of the code for HFA files
>> >> folded into it.
>> >>
>> >> The workstation is running Windows 7 64-bit w/ 8 GB of RAM and too
>> >> much disk space to know what to do with .. for the moment.
>> >>
>> >> Thanks.
>> >> = David
>> >>
>> >> ----- % cut here % -----
>> >> Driver: GTiff/GeoTIFF
>> >> Files: Central_North.tif
>> >> Size is 119999, 74401
>> >> Coordinate System is:
>> >> GEOGCS["WGS 84",
>> >>     DATUM["WGS_1984",
>> >>         SPHEROID["WGS 84",6378137,298.257223563,
>> >>             AUTHORITY["EPSG","7030"]],
>> >>         AUTHORITY["EPSG","6326"]],
>> >>     PRIMEM["Greenwich",0],
>> >>     UNIT["degree",0.0174532925199433],
>> >>     AUTHORITY["EPSG","4326"]]
>> >> Origin = (-31.999583000000001,60.000416333276846)
>> >> Pixel Size = (0.000833333333333,-0.000833333333333)
>> >> Metadata:
>> >>   AREA_OR_POINT=Area
>> >> Image Structure Metadata:
>> >>   COMPRESSION=LZW
>> >>   INTERLEAVE=BAND
>> >> Corner Coordinates:
>> >> Upper Left  ( -31.9995830,  60.0004163) ( 31d59'58.50"W, 60d 0'1.50"N)
>> >> Lower Left  ( -31.9995830,  -2.0004170) ( 31d59'58.50"W,  2d 0'1.50"S)
>> >> Upper Right (  67.9995837,  60.0004163) ( 67d59'58.50"E, 60d 0'1.50"N)
>> >> Lower Right (  67.9995837,  -2.0004170) ( 67d59'58.50"E,  2d 0'1.50"S)
>> >> Center      (  18.0000003,  28.9999997) ( 18d 0'0.00"E, 29d 0'0.00"N)
>> >> Band 1 Block=119999x1 Type=Int16, ColorInterp=Gray
>> >>   NoData Value=-32768
>> >>   Metadata:
>> >>     LAYER_TYPE=athematic
>> >> ----- % cut here % -----
>> >>
>> >> On Tue, Feb 9, 2010 at 2:09 PM, Even Rouault
>> >>
>> >> <even.rouault at mines-paris.org> wrote:
>> >> > Could you append the output of gdalinfo on the TIF you make overviews
>> >> > on and the exact gdaladdo you're using (mainly if you use one
>> >> > particular resampling algorithm) ?
>> >> >
>> >> > Le Tuesday 09 February 2010 22:57:26 David Fogel, vous avez écrit :
>> >> >> Hi:
>> >> >>
>> >> >> In the process of moving data into GeoTIFF, I am re-creating pyramids
>> >> >> (OVR files).  The good news: It takes fewer than ten minutes to
>> >> >> create an LZW Horizontal Differenced compressed 3.5 GB file on a
>> >> >> Windows 7 box using GDAL 1.70+ ( via Tamas Szekeres site; 64-bit
>> >> >> precompiled ).
>> >> >>
>> >> >> The OVR files are problematic.  Using gdal_translate, I can resize by
>> >> >> 50% in less than ten minutes.  Using gdaladdo, the first layer ( "2"
>> >> >> ) takes hours to complete.
>> >> >>
>> >> >> This is for SRTM 16-bit data.  It appears that something is amiss.
>> >> >>
>> >> >> Tips and tricks?
>> >> >>
>> >> >> Thanks!
>> >> >> = David
>> >> >> _______________________________________________
>> >> >> gdal-dev mailing list
>> >> >> gdal-dev at lists.osgeo.org
>> >> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>