[GRASS-dev] GRASS GIS raster files: LZW compression?

[Markus Metz]

On Thu, Oct 26, 2017 at 7:23 PM, Even Rouault <even.rouault at spatialys.com>
wrote:
>
> On jeudi 26 octobre 2017 18:57:10 CEST Markus Neteler wrote:
>
> > On Thu, Oct 26, 2017 at 9:25 AM, Markus Metz
>
> >
>
> > <markus.metz.giswork at gmail.com> wrote:
>
> > > On Wed, Oct 25, 2017 at 10:40 PM, Markus Neteler <neteler at osgeo.org>
wrote:
>
> > >> Hi devs,
>
> > >>
>
> > >> while playing with the SRTM 30m world I see that there is a
difference
>
> > >> of 66GB in size between LZ4 (larger) and DEFLATE (smaller). While
this
>
> > >> is not a surprise I was wondering if LZW compression could be brought
>
> > >> back. AFAIK the license issue is gone for a while and indeed GRASS
got
>
> > >> a special permit back in 1988 according to the old source code.
>
> > >>
>
> > >> Would that make sense?
>
> > >
>
> > > You could test with compress (LZW) and gzip (DEFLATE based on LZ77) if
>
> > > that
>
> > > would make sense.
>
> > >
>
> > > Implementation would be a bit of an effort: either adapt from the
source
>
> > > code of compress or from GDAL's LZW compression for Tiff.
>
> >
>
> > I thought to remember that the LZW compression was in libgis some
>
> > decades ago but did not find any trace so far.
>
> > At that point I happily abandon this idea - no need to waste energy
here :-)
>
>
>
> Another option would be to consider the zstd / zstandard compression
method
>
> ( https://github.com/facebook/zstd ), that has roughly the compression
ratio of DEFLATE / LZ77 with a compression and decompression speed 3 or 4
times faster. This is in my todo list of things to experiment as a libtiff
compression method.
>
>
>
> Zstd has been added recently as a compression method in the Linux kernel:
>
>
https://www.phoronix.com/scan.php?page=news_item&px=ZSTD-Linux-Kernel-Image
>
Strong arguments for zstd! It seems that zstd is also adopted by other
projects, and also available for RHEL 7 via EPEL.

Some quick test gives me zstd compression speed a bit slower than lz4 and
zstd compression ratio a bit better than zlib's deflate, i.e. with about
the speed of lz4 you can get compression ratio a bit better than deflate,
that's great! Regarding compression ratio, zstd with ultra settings can get
close to bzip2, but is then much slower than bzip2 with zstd's memory usage
exploding.

The zstd API is similar to the API of zlib/lz4/bzip2, that means adding
zstd to GRASS could be done by cloning e.g. the GRASS interface to lz4.

Markus M
>
>
> Even
>
>
>
> --
>
> Spatialys - Geospatial professional services
>
> http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20171027/9d36a62c/attachment.html>