[GRASS-dev] GRASS GIS raster files: LZW compression?

[Markus Metz]

ZSTD compression has been added to trunk with r71889-92.

ZSTD compression can be added with configure --with-zstd=yes or simpler
configure --with-zstd

In order to get some wider testing, ZSTD is now the default compression
method in trunk if ZSTD is available.

ZSTD is an improvement over ZLIB: generally it compresses faster and higher
than ZLIB. Decompression is consistently much faster than with ZLIB. For
more information on ZSTD (Zstandard), see
http://facebook.github.io/zstd/
https://github.com/facebook/zstd

Please test!

Markus M


On Sat, Dec 2, 2017 at 12:35 PM, Markus Neteler <neteler at osgeo.org> wrote:

> On Tue, Oct 31, 2017 at 8:35 PM, Markus Metz
> <markus.metz.giswork at gmail.com> wrote:
> > On Tue, Oct 31, 2017 at 12:16 PM, Markus Neteler <neteler at osgeo.org>
> wrote:
> >>
> >> On Mon, Oct 30, 2017 at 9:58 AM, Markus Metz
> >> <markus.metz.giswork at gmail.com> wrote:
> >> ...
> >> > Some more tests:
> >> > corine land cover 250 meter, relatively small, easy to compress Byte
> >> > type
> >> > LZ4: 33 sec, 54 MB
> >> > ZLIB level 1: 36 sec, 39 MB
> >> > ZSTD level 3: 34 sec, 29 MB
> >> >
> >> > GMTED2010 for the corine region as float, difficult to compress
> >> > LZ4: 27 sec, 3.1 GB
> >> > ZLIB level 1: 129 sec, 2.7 GB
> >> > ZSTD level 3: 38 sec, 2.8 GB
> >> >
> >> > GMTED2010 for the corine region as integer, easy to compress but a lot
> >> > of
> >> > data
> >> > LZ4: 38 sec, 1.4 GB
> >> > ZLIB level 1: 73 sec, 993 MB
> >> > ZSTD level 3: 53 sec, 982 MB
> >> >
> >> > attempting maximum compression of GMTED2010 as integer
> >> > ZLIB level 9: 735 sec, 945 MB
> >> > ZSTD level 19: 485 sec, 846 MB
> >> > BZIP2: 273 sec, 618 MB
> >> >
> >> > With easy to compress data, ZSTD is faster and compresses better than
> >> > ZLIB.
> >> > With difficult to compress data (floating point), ZSTD is at least 3x
> >> > faster
> >> > than ZLIB with slightly worse compression than ZLIB, but still better
> >> > than
> >> > LZ4.
> >>
> >> Thanks for these comparisons! Quite interesting (especially for
> >> datasets which need to be stored on expensive SSDs).
> >
> > Also interesting for a GRASS database on a remote cloud (i.e. slow data
> > transfer).
> >>
> >> >> The zstd API is similar to the API of zlib/lz4/bzip2, that means
> adding
> >> >> zstd to GRASS could be done by cloning e.g. the GRASS interface to
> lz4.
> >> >
> >> > ZSTD seems to be an improvement over ZLIB: it is always faster and
> most
> >> > of
> >> > the time compression ratio is higher. I see the biggest advantage for
> >> > difficult to compress data where ZSTD is considerably faster than
> ZLIB,
> >> > with
> >> > similar compression. Therefore I think adding ZSTD to GRASS would be a
> >> > good
> >> > idea.
> >>
> >> It looks very promising (how much effort would that be?).
> >
> > basically the same like for bzip2
> >
> > 1. add ZSTD to configure, config.h.in, Platform.make.in
> > 2. link gislib against ZSTD if requested and available
> > 3. add the interfaces to ZSTD in a new cmprzstd.c
> > 4. test that everything is working fine and that ZSTD in GRASS is indeed
> an
> > improvement over ZLIB.
> >
> > alltogether 4-5 days I guess.
>
> ok, we'll fund that development :-)
>
> markusN
>
>
> --
> Markus Neteler, PhD
> http://www.mundialis.de - free data with free software
> http://grass.osgeo.org
> http://courses.neteler.org/blog
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20171204/d3f8f000/attachment.html>