[gdal-dev] Question on TIFF compression alternatives

[Neumann, Andreas]

Hi Even, 

Thank you for your detailed reply. 

I should have added that my concern is mainly about random access to
parts of the images - for QGIS server and QGIS Desktop. 

I have internal pyramids. I am not so sure if it is tiled or not. Does
gdalinfo reveal whether the TIFF was created with "TILED=YES"? 

So from your analysis it looks like lzma is the slowest, but overall,
the differences aren't that huge. I don't know how random access to
parts of the image changes the game a bit, compared to your analysis
where you examine the decompression of the whole image - right? 

Thanks, 

Andreas 

On 2016-10-14 12:09, Even Rouault wrote:

> Le vendredi 14 octobre 2016 09:45:38, Neumann, Andreas a écrit : 
> 
>> Hi,
>> 
>> I have a question on TIFF compression alternatives.
>> 
>> My data of concern is 8bit gray value. Lots of white pixel (>95%), the
>> rest is black pixels (content) and some gray pixels because of
>> antialiasing.
>> 
>> I would like to compress to save some storage, but I need a good
>> performance. What is the recommended compression that is fast on
>> decompression?
>> 
>> I tried DEFLATE, but it is rather slow to decompress. Would LZW or
>> PACKBITS be preferred in this situation? How about LZMA? I would like to
>> avoid non-compressed data, because I use SSD storage for performance
>> reasons.
> 
> My findings on a 37614 x 18816  8 bit image (a scan of a text page) that should 
> be similar to what you describe.
> 
> Decompression time is the result of the second run of "time gdalinfo -
> checksum" (so you can consider that I/O is cached by the OS and you measure 
> essentially CPU time. storage is spinning disk)
> 
> No tiling:
> 
> Size            Compression            Decompression time
> 707895750    out_nocompression.tif    9.3 s
> 
> 11239972    out_deflate.tif            10.8 s
> 17749306    out_lzw.tif                10.5 s
> 12337710    out_lzma.tif            12.6 s
> 36201320    out_packbits.tif            9.9 s
> 
> Tiling:
> 
> Size            Compression            Decompression time
> 8813833        out_deflate_tiled.tif        11.1 s
> 14614121    out_lzw_tiled.tif            11.0 s
> 8479238        out_lzma_tiled.tif        13.0 s
> 37018470    out_packbits_tiled.tif        10.2 s
> 
> Note: in the tiling case, the gdalinfo -checksum benchmark might not be ideal 
> as it proceeds line by line (and not block by block), thus there's a lot of 
> copying between cached blocks and output buffer that is done.
> 
> So I've done time gdal_translate XXX.tif /vsimem/out.tif -q :
> 
> deflate tiled:    2.5 s
> lzw tiled:        2.3 s
> packbits tiled:     1.5 s
> lzma tiled:        4.5 s
> 
> Times for non tiled files are essentially the same.
> 
> Note: if you use tiling and you have whole tiles that are at white value, if 
> you use GDAL trunk, set the SPARSE_OK=YES creation option and set the nodata 
> value to 255, then those tiles will be entirely omitted from the compressed 
> file. This might save a few extra bytes, and also some decompression time.
> 
> gdal_translate out_nocompression.tif out_packbits_tiled.tif -co tiled=yes  -co 
> compress=packbits
> gdal_translate out_nocompression.tif out_packbits_tiled_sparse.tif -co 
> tiled=yes -a_nodata 255 -co compress=packbits -co sparse_ok=yes
> 
> 37018470    out_packbits_tiled.tif
> 30409074    out_packbits_tiled_sparse.tif
> 
> time gdal_translate out_packbits_tiled_sparse.tif /vsimem/out.tif -q: 1.3 s
> 
> Even

  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20161014/f331cdff/attachment.html>