[gdal-dev] Question on TIFF compression alternatives

[Even Rouault]

Le vendredi 14 octobre 2016 13:01:50, Neumann, Andreas a écrit :
> Hi Even,
> 
> Thank you for your detailed reply.
> 
> I should have added that my concern is mainly about random access to
> parts of the images - for QGIS server and QGIS Desktop.
> 
> I have internal pyramids. I am not so sure if it is tiled or not.

Internal pyramids created by GDAL are tiled with tiles of 128x128 pixel (can 
be changed with the GDAL_TIFF_OVR_BLOCKSIZE configuration option: 
http://gdal.org/frmt_gtiff.html#overviews . But the default is generally fine)

> Does
> gdalinfo reveal whether the TIFF was created with "TILED=YES"?

If you see something "Band 1 Block=256x256" in the gdalinfo output, the full 
resolution image is tiled. If the block width (the first figure after Block=) is 
the image width, then it is not. You definitely want to use a tiled image for 
random access.

> 
> So from your analysis it looks like lzma is the slowest, but overall,
> the differences aren't that huge. I don't know how random access to
> parts of the image changes the game a bit, compared to your analysis
> where you examine the decompression of the whole image - right?

I used whole image decompression otherwise the timings would have been hard to 
measure accurately. You can expect the same ratios of performance to apply to 
partial image decompression.

My timing were done on "hot" runs. Depending on use cases, I/O might be the 
overwhelming time.

> 
> Thanks,
> 
> Andreas
> 
> On 2016-10-14 12:09, Even Rouault wrote:
> > Le vendredi 14 octobre 2016 09:45:38, Neumann, Andreas a écrit :
> >> Hi,
> >> 
> >> I have a question on TIFF compression alternatives.
> >> 
> >> My data of concern is 8bit gray value. Lots of white pixel (>95%), the
> >> rest is black pixels (content) and some gray pixels because of
> >> antialiasing.
> >> 
> >> I would like to compress to save some storage, but I need a good
> >> performance. What is the recommended compression that is fast on
> >> decompression?
> >> 
> >> I tried DEFLATE, but it is rather slow to decompress. Would LZW or
> >> PACKBITS be preferred in this situation? How about LZMA? I would like to
> >> avoid non-compressed data, because I use SSD storage for performance
> >> reasons.
> > 
> > My findings on a 37614 x 18816  8 bit image (a scan of a text page) that
> > should be similar to what you describe.
> > 
> > Decompression time is the result of the second run of "time gdalinfo -
> > checksum" (so you can consider that I/O is cached by the OS and you
> > measure essentially CPU time. storage is spinning disk)
> > 
> > No tiling:
> > 
> > Size            Compression            Decompression time
> > 707895750    out_nocompression.tif    9.3 s
> > 
> > 11239972    out_deflate.tif            10.8 s
> > 17749306    out_lzw.tif                10.5 s
> > 12337710    out_lzma.tif            12.6 s
> > 36201320    out_packbits.tif            9.9 s
> > 
> > Tiling:
> > 
> > Size            Compression            Decompression time
> > 8813833        out_deflate_tiled.tif        11.1 s
> > 14614121    out_lzw_tiled.tif            11.0 s
> > 8479238        out_lzma_tiled.tif        13.0 s
> > 37018470    out_packbits_tiled.tif        10.2 s
> > 
> > Note: in the tiling case, the gdalinfo -checksum benchmark might not be
> > ideal as it proceeds line by line (and not block by block), thus there's
> > a lot of copying between cached blocks and output buffer that is done.
> > 
> > So I've done time gdal_translate XXX.tif /vsimem/out.tif -q :
> > 
> > deflate tiled:    2.5 s
> > lzw tiled:        2.3 s
> > packbits tiled:     1.5 s
> > lzma tiled:        4.5 s
> > 
> > Times for non tiled files are essentially the same.
> > 
> > Note: if you use tiling and you have whole tiles that are at white value,
> > if you use GDAL trunk, set the SPARSE_OK=YES creation option and set the
> > nodata value to 255, then those tiles will be entirely omitted from the
> > compressed file. This might save a few extra bytes, and also some
> > decompression time.
> > 
> > gdal_translate out_nocompression.tif out_packbits_tiled.tif -co tiled=yes
> >  -co compress=packbits
> > gdal_translate out_nocompression.tif out_packbits_tiled_sparse.tif -co
> > tiled=yes -a_nodata 255 -co compress=packbits -co sparse_ok=yes
> > 
> > 37018470    out_packbits_tiled.tif
> > 30409074    out_packbits_tiled_sparse.tif
> > 
> > time gdal_translate out_packbits_tiled_sparse.tif /vsimem/out.tif -q: 1.3
> > s
> > 
> > Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com