[GRASS-dev] GRASS GIS raster files: LZW compression?

[Markus Metz]

On Tue, Oct 31, 2017 at 12:16 PM, Markus Neteler <neteler at osgeo.org> wrote:
>
> On Mon, Oct 30, 2017 at 9:58 AM, Markus Metz
> <markus.metz.giswork at gmail.com> wrote:
> ...
> > Some more tests:
> > corine land cover 250 meter, relatively small, easy to compress Byte
type
> > LZ4: 33 sec, 54 MB
> > ZLIB level 1: 36 sec, 39 MB
> > ZSTD level 3: 34 sec, 29 MB
> >
> > GMTED2010 for the corine region as float, difficult to compress
> > LZ4: 27 sec, 3.1 GB
> > ZLIB level 1: 129 sec, 2.7 GB
> > ZSTD level 3: 38 sec, 2.8 GB
> >
> > GMTED2010 for the corine region as integer, easy to compress but a lot
of
> > data
> > LZ4: 38 sec, 1.4 GB
> > ZLIB level 1: 73 sec, 993 MB
> > ZSTD level 3: 53 sec, 982 MB
> >
> > attempting maximum compression of GMTED2010 as integer
> > ZLIB level 9: 735 sec, 945 MB
> > ZSTD level 19: 485 sec, 846 MB
> > BZIP2: 273 sec, 618 MB
> >
> > With easy to compress data, ZSTD is faster and compresses better than
ZLIB.
> > With difficult to compress data (floating point), ZSTD is at least 3x
faster
> > than ZLIB with slightly worse compression than ZLIB, but still better
than
> > LZ4.
>
> Thanks for these comparisons! Quite interesting (especially for
> datasets which need to be stored on expensive SSDs).

Also interesting for a GRASS database on a remote cloud (i.e. slow data
transfer).
>
> >> The zstd API is similar to the API of zlib/lz4/bzip2, that means adding
> >> zstd to GRASS could be done by cloning e.g. the GRASS interface to lz4.
> >
> > ZSTD seems to be an improvement over ZLIB: it is always faster and most
of
> > the time compression ratio is higher. I see the biggest advantage for
> > difficult to compress data where ZSTD is considerably faster than ZLIB,
with
> > similar compression. Therefore I think adding ZSTD to GRASS would be a
good
> > idea.
>
> It looks very promising (how much effort would that be?).

basically the same like for bzip2

1. add ZSTD to configure, config.h.in, Platform.make.in
2. link gislib against ZSTD if requested and available
3. add the interfaces to ZSTD in a new cmprzstd.c
4. test that everything is working fine and that ZSTD in GRASS is indeed an
improvement over ZLIB.

alltogether 4-5 days I guess.
>
> ...looks like becoming more widespread: http://www.zstd.net

Indeed. It is also available for RHEL 7 and its derivates (important
production systems) through EPEL.

Also, the authors seem to be quite confident about it, calling it Z
standard, the new standard for LZ77 compression.

Markus M

>
> markusN
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20171031/d3be4bb3/attachment.html>