[GRASS-dev] GRASS GIS raster files: LZW compression?

[Markus Neteler]

On Tue, Oct 31, 2017 at 8:35 PM, Markus Metz
<markus.metz.giswork at gmail.com> wrote:
> On Tue, Oct 31, 2017 at 12:16 PM, Markus Neteler <neteler at osgeo.org> wrote:
>>
>> On Mon, Oct 30, 2017 at 9:58 AM, Markus Metz
>> <markus.metz.giswork at gmail.com> wrote:
>> ...
>> > Some more tests:
>> > corine land cover 250 meter, relatively small, easy to compress Byte
>> > type
>> > LZ4: 33 sec, 54 MB
>> > ZLIB level 1: 36 sec, 39 MB
>> > ZSTD level 3: 34 sec, 29 MB
>> >
>> > GMTED2010 for the corine region as float, difficult to compress
>> > LZ4: 27 sec, 3.1 GB
>> > ZLIB level 1: 129 sec, 2.7 GB
>> > ZSTD level 3: 38 sec, 2.8 GB
>> >
>> > GMTED2010 for the corine region as integer, easy to compress but a lot
>> > of
>> > data
>> > LZ4: 38 sec, 1.4 GB
>> > ZLIB level 1: 73 sec, 993 MB
>> > ZSTD level 3: 53 sec, 982 MB
>> >
>> > attempting maximum compression of GMTED2010 as integer
>> > ZLIB level 9: 735 sec, 945 MB
>> > ZSTD level 19: 485 sec, 846 MB
>> > BZIP2: 273 sec, 618 MB
>> >
>> > With easy to compress data, ZSTD is faster and compresses better than
>> > ZLIB.
>> > With difficult to compress data (floating point), ZSTD is at least 3x
>> > faster
>> > than ZLIB with slightly worse compression than ZLIB, but still better
>> > than
>> > LZ4.
>>
>> Thanks for these comparisons! Quite interesting (especially for
>> datasets which need to be stored on expensive SSDs).
>
> Also interesting for a GRASS database on a remote cloud (i.e. slow data
> transfer).
>>
>> >> The zstd API is similar to the API of zlib/lz4/bzip2, that means adding
>> >> zstd to GRASS could be done by cloning e.g. the GRASS interface to lz4.
>> >
>> > ZSTD seems to be an improvement over ZLIB: it is always faster and most
>> > of
>> > the time compression ratio is higher. I see the biggest advantage for
>> > difficult to compress data where ZSTD is considerably faster than ZLIB,
>> > with
>> > similar compression. Therefore I think adding ZSTD to GRASS would be a
>> > good
>> > idea.
>>
>> It looks very promising (how much effort would that be?).
>
> basically the same like for bzip2
>
> 1. add ZSTD to configure, config.h.in, Platform.make.in
> 2. link gislib against ZSTD if requested and available
> 3. add the interfaces to ZSTD in a new cmprzstd.c
> 4. test that everything is working fine and that ZSTD in GRASS is indeed an
> improvement over ZLIB.
>
> alltogether 4-5 days I guess.

ok, we'll fund that development :-)

markusN


-- 
Markus Neteler, PhD
http://www.mundialis.de - free data with free software
http://grass.osgeo.org
http://courses.neteler.org/blog