[gdal-dev] Question on TIFF compression alternatives

[Even Rouault]

Le vendredi 14 octobre 2016 09:45:38, Neumann, Andreas a écrit :
> Hi,
> 
> I have a question on TIFF compression alternatives.
> 
> My data of concern is 8bit gray value. Lots of white pixel (>95%), the
> rest is black pixels (content) and some gray pixels because of
> antialiasing.
> 
> I would like to compress to save some storage, but I need a good
> performance. What is the recommended compression that is fast on
> decompression?
> 
> I tried DEFLATE, but it is rather slow to decompress. Would LZW or
> PACKBITS be preferred in this situation? How about LZMA? I would like to
> avoid non-compressed data, because I use SSD storage for performance
> reasons.

My findings on a 37614 x 18816  8 bit image (a scan of a text page) that should 
be similar to what you describe.

Decompression time is the result of the second run of "time gdalinfo -
checksum" (so you can consider that I/O is cached by the OS and you measure 
essentially CPU time. storage is spinning disk)

No tiling:

Size			Compression			Decompression time
707895750	out_nocompression.tif	9.3 s

11239972	out_deflate.tif			10.8 s
17749306	out_lzw.tif				10.5 s
12337710	out_lzma.tif			12.6 s
36201320	out_packbits.tif			9.9 s

Tiling:

Size			Compression			Decompression time
 8813833		out_deflate_tiled.tif		11.1 s
14614121	out_lzw_tiled.tif			11.0 s
 8479238		out_lzma_tiled.tif		13.0 s
37018470	out_packbits_tiled.tif		10.2 s

Note: in the tiling case, the gdalinfo -checksum benchmark might not be ideal 
as it proceeds line by line (and not block by block), thus there's a lot of 
copying between cached blocks and output buffer that is done.

So I've done time gdal_translate XXX.tif /vsimem/out.tif -q :

deflate tiled:	2.5 s
lzw tiled:		2.3 s
packbits tiled: 	1.5 s
lzma tiled:		4.5 s

Times for non tiled files are essentially the same.


Note: if you use tiling and you have whole tiles that are at white value, if 
you use GDAL trunk, set the SPARSE_OK=YES creation option and set the nodata 
value to 255, then those tiles will be entirely omitted from the compressed 
file. This might save a few extra bytes, and also some decompression time.

gdal_translate out_nocompression.tif out_packbits_tiled.tif -co tiled=yes  -co 
compress=packbits
gdal_translate out_nocompression.tif out_packbits_tiled_sparse.tif -co 
tiled=yes -a_nodata 255 -co compress=packbits -co sparse_ok=yes

37018470	out_packbits_tiled.tif
30409074	out_packbits_tiled_sparse.tif

time gdal_translate out_packbits_tiled_sparse.tif /vsimem/out.tif -q: 1.3 s

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com