[GRASS-dev] GRASS GIS raster files: LZW compression?

[Markus Neteler]

On Mon, Oct 30, 2017 at 9:58 AM, Markus Metz
<markus.metz.giswork at gmail.com> wrote:
...
> Some more tests:
> corine land cover 250 meter, relatively small, easy to compress Byte type
> LZ4: 33 sec, 54 MB
> ZLIB level 1: 36 sec, 39 MB
> ZSTD level 3: 34 sec, 29 MB
>
> GMTED2010 for the corine region as float, difficult to compress
> LZ4: 27 sec, 3.1 GB
> ZLIB level 1: 129 sec, 2.7 GB
> ZSTD level 3: 38 sec, 2.8 GB
>
> GMTED2010 for the corine region as integer, easy to compress but a lot of
> data
> LZ4: 38 sec, 1.4 GB
> ZLIB level 1: 73 sec, 993 MB
> ZSTD level 3: 53 sec, 982 MB
>
> attempting maximum compression of GMTED2010 as integer
> ZLIB level 9: 735 sec, 945 MB
> ZSTD level 19: 485 sec, 846 MB
> BZIP2: 273 sec, 618 MB
>
> With easy to compress data, ZSTD is faster and compresses better than ZLIB.
> With difficult to compress data (floating point), ZSTD is at least 3x faster
> than ZLIB with slightly worse compression than ZLIB, but still better than
> LZ4.

Thanks for these comparisons! Quite interesting (especially for
datasets which need to be stored on expensive SSDs).

>> The zstd API is similar to the API of zlib/lz4/bzip2, that means adding
>> zstd to GRASS could be done by cloning e.g. the GRASS interface to lz4.
>
> ZSTD seems to be an improvement over ZLIB: it is always faster and most of
> the time compression ratio is higher. I see the biggest advantage for
> difficult to compress data where ZSTD is considerably faster than ZLIB, with
> similar compression. Therefore I think adding ZSTD to GRASS would be a good
> idea.

It looks very promising (how much effort would that be?).

...looks like becoming more widespread: http://www.zstd.net

markusN