[GRASS-dev] GRASS GIS raster files: LZW compression?

Markus Metz markus.metz.giswork at gmail.com
Mon Oct 30 01:58:55 PDT 2017


On Fri, Oct 27, 2017 at 10:44 PM, Markus Metz <markus.metz.giswork at gmail.com>
wrote:
>
>
>
> On Thu, Oct 26, 2017 at 7:23 PM, Even Rouault <even.rouault at spatialys.com>
wrote:
> >
> > On jeudi 26 octobre 2017 18:57:10 CEST Markus Neteler wrote:
> >
> > > On Thu, Oct 26, 2017 at 9:25 AM, Markus Metz
> >
> > >
> >
> > > <markus.metz.giswork at gmail.com> wrote:
> >
> > > > On Wed, Oct 25, 2017 at 10:40 PM, Markus Neteler <neteler at osgeo.org>
wrote:
> >
> > > >> Hi devs,
> >
> > > >>
> >
> > > >> while playing with the SRTM 30m world I see that there is a
difference
> >
> > > >> of 66GB in size between LZ4 (larger) and DEFLATE (smaller). While
this
> >
> > > >> is not a surprise I was wondering if LZW compression could be
brought
> >
> > > >> back. AFAIK the license issue is gone for a while and indeed GRASS
got
> >
> > > >> a special permit back in 1988 according to the old source code.
> >
> > > >>
> >
> > > >> Would that make sense?
> >
> > > >
> >
> > > > You could test with compress (LZW) and gzip (DEFLATE based on LZ77)
if
> >
> > > > that
> >
> > > > would make sense.
> >
> > > >
> >
> > > > Implementation would be a bit of an effort: either adapt from the
source
> >
> > > > code of compress or from GDAL's LZW compression for Tiff.
> >
> > >
> >
> > > I thought to remember that the LZW compression was in libgis some
> >
> > > decades ago but did not find any trace so far.
> >
> > > At that point I happily abandon this idea - no need to waste energy
here :-)
> >
> >
> >
> > Another option would be to consider the zstd / zstandard compression
method
> >
> > ( https://github.com/facebook/zstd ), that has roughly the compression
ratio of DEFLATE / LZ77 with a compression and decompression speed 3 or 4
times faster. This is in my todo list of things to experiment as a libtiff
compression method.
> >
> >
> >
> > Zstd has been added recently as a compression method in the Linux
kernel:
> >
> >
https://www.phoronix.com/scan.php?page=news_item&px=ZSTD-Linux-Kernel-Image
> >
> Strong arguments for zstd! It seems that zstd is also adopted by other
projects, and also available for RHEL 7 via EPEL.
>
> Some quick test gives me zstd compression speed a bit slower than lz4 and
zstd compression ratio a bit better than zlib's deflate, i.e. with about
the speed of lz4 you can get compression ratio a bit better than deflate,
that's great! Regarding compression ratio, zstd with ultra settings can get
close to bzip2, but is then much slower than bzip2 with zstd's memory usage
exploding.

Some more tests:
corine land cover 250 meter, relatively small, easy to compress Byte type
LZ4: 33 sec, 54 MB
ZLIB level 1: 36 sec, 39 MB
ZSTD level 3: 34 sec, 29 MB

GMTED2010 for the corine region as float, difficult to compress
LZ4: 27 sec, 3.1 GB
ZLIB level 1: 129 sec, 2.7 GB
ZSTD level 3: 38 sec, 2.8 GB

GMTED2010 for the corine region as integer, easy to compress but a lot of
data
LZ4: 38 sec, 1.4 GB
ZLIB level 1: 73 sec, 993 MB
ZSTD level 3: 53 sec, 982 MB

attempting maximum compression of GMTED2010 as integer
ZLIB level 9: 735 sec, 945 MB
ZSTD level 19: 485 sec, 846 MB
BZIP2: 273 sec, 618 MB

With easy to compress data, ZSTD is faster and compresses better than ZLIB.
With difficult to compress data (floating point), ZSTD is at least 3x
faster than ZLIB with slightly worse compression than ZLIB, but still
better than LZ4.

>
> The zstd API is similar to the API of zlib/lz4/bzip2, that means adding
zstd to GRASS could be done by cloning e.g. the GRASS interface to lz4.

ZSTD seems to be an improvement over ZLIB: it is always faster and most of
the time compression ratio is higher. I see the biggest advantage for
difficult to compress data where ZSTD is considerably faster than ZLIB,
with similar compression. Therefore I think adding ZSTD to GRASS would be a
good idea.

Markus M
> >
> >
> > Even
> >
> >
> >
> > --
> >
> > Spatialys - Geospatial professional services
> >
> > http://www.spatialys.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20171030/15a6fbb3/attachment.html>


More information about the grass-dev mailing list