[gdal-dev] Read data without reading 'null data'

[Even Rouault]

Le vendredi 19 février 2016 10:48:50, jramm a écrit :
> We are storing large raster datasets (~500,000 x 500,000) in geotiff files.
> The array data is typically very sparse (density is 0.02% or less) and
> compression greatly reduces the geotiff size.
> 
> However, when processing data we read the data in chunks, which are
> automatically decompressed..this means we are often reading in large areas
> of no data and 'skipping' them.
> Is there a way to only read data with values so we may apply the processing
> to these cells, and write them back out, skipping the no data? I guess this
> would be equivalent to just not decompressing?

Hi,

The GTiff driver supports the concept of "sparse" files, which is I think a 
GDAL-specific extension, of how tiles and strips are indexed in TIFF. Basically 
if the byte size of a given block is marked as 0, it is interpreted as not 
physically present in the file and, when the block is read through the GDAL 
API, the pixels are initialized to 0 or the nodata value.

Currently sparse files can only be generated with the Create() API with the 
SPARSE_OK creation option, and by not writting any data (not even 0 or the 
nodata value) in the blocks you don't want to be serialized. Consequently 
using gdal_translate or gdalwarp will not produce the desired effect. Although 
that could be certainly improved.

On the reading side, in recent releases (2.0 I think), you can query the 
BLOCK_SIZE_{blockxoff}_{blockyoff} metadata item in the TIFF metadata domain to 
know if a block is present or not (return will be NULL for absent)

Even

> 
> 
> 
> --
> View this message in context:
> http://osgeo-org.1560.x6.nabble.com/Read-data-without-reading-null-data-tp
> 5251626.html Sent from the GDAL - Dev mailing list archive at Nabble.com.
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com