[gdal-dev] Compression not applying when updating/exiting GeoTiff file in GDAL?

[Even Rouault]

On vendredi 29 mars 2019 09:02:08 CET taylorday.assimila at gmail.com wrote:
> Hi all,
> 
> 
> 
> I posted this question on GIS SE (link
> <https://gis.stackexchange.com/questions/317009/compression-not-applying-whe
> n-updating-exiting-geotiff-file-in-gdal?noredirect=1#comment515654_317009>
> ) but have not had any luck, so was wondering if anyone here could help.
> 
> 
> 
> I have a `GeoTiff` file with dimensions 300 x 300 and 8760 bands (hourly for
> one full year). I am testing some code that opens the file, updates ONE
> band and exits - but I am finding that the file size almost doubles when I
> exit the file, despite the new band being the same data type etc. Code is
> as follows:
> 
> 
> 
> I am creating the `GeoTiff` file using the following creation options using
> many singular `GeoTiff` files:
> 
> 
> 
>     gdal_merge.py -separate -co BIGTIFF=YES -co COMPRESS=DEFLATE -co
> PREDICTOR=1 -co TILED=YES -co BLOCKXSIZE=64 -co BLOCKYSIZE=64 -o block64.tif
> *.tif

This will create a geotiff file with default pixel-interleaving. If you want 
to update bands separately, you might prefer adding -co INTERLEAVE=BAND, so 
that TIFF tiles only contain data for one single band. With the default 
INTERLEAVE=PIXEL, when you rewrite a single band, you must be aware that you 
end up rewriting the whole file under the hood.
Which explains the dramatic effect you see on file size currently. One thing 
to have in mind is that the libtiff library used by GDAL has no "garbage 
collection", so when using compression, if the updated compressed data 
occupies a larger space than the previous one, the old space is lost and the 
new compressed data is written at the end of the file.

I don't know the datatype you use, but a uncompressed file could also be an 
option and won't have issues with growing size when being updated. This would 
be at worst a 7.1 GB file for 64-bit floating point data type.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com