[GRASS-dev] GRASS GIS raster files: LZW compression?

[Markus Metz]

On Tue, Dec 5, 2017 at 10:55 AM, Stefan Blumentrath <
Stefan.Blumentrath at nina.no> wrote:
>
> Great indeed!
> Any chance to get that in 7.4.1?
> Or would we have to wait for 7.6 to utilize this in production?

ZSTD compression is a new feature and the changes are not trivial,
considering the changesets for r71889, r71890, r71891. There is a minimal
chance to get ZSTD compression in 7.4. GRASS 7.4.0 is not yet out,
therefore ZSTD might go to there, at the risk of slowing down the release
of GRASS 7.4.0. Any subsequent 7.4.x releases will be only bug-fix
releases. GRASS 7.4.0 should get out in time to make it into the next
releases of major Linux distros.

Markus M

>
> Cheers
> Stefan
>
> -----Original Message-----
> From: grass-dev [mailto:grass-dev-bounces at lists.osgeo.org] On Behalf Of
Markus Neteler
> Sent: tirsdag 5. desember 2017 10.08
> To: Markus Metz <markus.metz.giswork at gmail.com>
> Cc: GRASS developers list <grass-dev at lists.osgeo.org>
> Subject: Re: [GRASS-dev] GRASS GIS raster files: LZW compression?
>
> On Mon, Dec 4, 2017 at 7:32 PM, Markus Metz <markus.metz.giswork at gmail.com>
wrote:
> > ZSTD compression has been added to trunk with r71889-92.
>
> Great!
>
> > ZSTD compression can be added with configure --with-zstd=yes or
> > simpler configure --with-zstd
> >
> > In order to get some wider testing, ZSTD is now the default
> > compression method in trunk if ZSTD is available.
> >
> > ZSTD is an improvement over ZLIB: generally it compresses faster and
> > higher than ZLIB. Decompression is consistently much faster than with
> > ZLIB. For more information on ZSTD (Zstandard), see
> > http://facebook.github.io/zstd/ https://github.com/facebook/zstd
> >
> > Please test!
>
> Here an initial test with a WorldClim DECLL map:
>
>
> GRASS 7.5.svn (latlong_wgs84): >
>
> g.region raster=tmean_12 -p
> projection: 3 (Latitude-Longitude)
> zone:       0
> datum:      wgs84
> ellipsoid:  wgs84
> north:      90N
> south:      60S
> west:       180W
> east:       180E
> nsres:      0:00:30
> ewres:      0:00:30
> rows:       18000
> cols:       43200
> cells:      777600000
>
> r.compress -p tmean_12
> <tmean_12> is compressed (method 2: ZLIB). Data type: DCELL <tmean_12>
has a compressed NULL file
>
>
> # create a ZSTD compressed copy:
> export GRASS_COMPRESS_NULLS=1
> export GRASS_COMPRESSOR=ZSTD
>
> r.mapcalc "tmean_12_ZSTD = tmean_12"
>
> r.compress -p tmean_12_ZSTD
> <tmean_12_ZSTD> is compressed (method 5: ZSTD). Data type: DCELL
<tmean_12_ZSTD> has a compressed NULL file
>
> # run r.univar
> time -p r.univar -g tmean_12
> n=222262507
> null_cells=555337493
> cells=777600000
> min=-51.9
> max=33.3
> range=85.2
> mean=-0.434087630455532
> mean_of_abs=18.7414294225023
> stddev=21.0312119498642
> variance=442.31187608011
> coeff_var=-4844.92311559167
> sum=-96481405.002736
> real 20.43
> user 20.21
> sys 0.11
>
> # comparison (results & timing)
> time -p r.univar -g tmean_12_ZSTD
> n=222262507
> null_cells=555337493
> cells=777600000
> min=-51.9
> max=33.3
> range=85.2
> mean=-0.434087630455532
> mean_of_abs=18.7414294225023
> stddev=21.0312119498642
> variance=442.31187608011
> coeff_var=-4844.92311559167
> sum=-96481405.002736
> real 15.43
> user 15.31
> sys 0.07
>
> # expectedly, the ZSTD is faster and it takes r.univar only ~ 75%
compared to ZLIB:
> > 15.43/20.43
> [1] 0.7552619
>
> ####
> # file sizes comparison:
> find . -name tmean_12 | sort | xargs du -h
> 4.0K    ./cats/tmean_12
> 4.0K    ./cellhd/tmean_12
> 2.7M    ./cell_misc/tmean_12
> 0    ./cell/tmean_12
> 12K    ./colr/tmean_12
> 256M    ./fcell/tmean_12
> 4.0K    ./hist/tmean_12
>
> find . -name tmean_12_ZSTD | sort | xargs du -h
> 4.0K    ./cats/tmean_12_ZSTD
> 4.0K    ./cellhd/tmean_12_ZSTD
> 2.7M    ./cell_misc/tmean_12_ZSTD
> 0    ./cell/tmean_12_ZSTD
> 12K    ./colr/tmean_12_ZSTD
> 195M    ./fcell/tmean_12_ZSTD
> 4.0K    ./hist/tmean_12_ZSTD
>
> # fcell size comparison in detail, here ZSTD is of ~75% smaller file
compared to ZLIB:
> > 203521319/268155000
> [1] 0.758969
>
>
> # my system: Fedora 26, 64bit, SSD disk, Intel(R) Core(TM) i5-6300U CPU @
2.40GHz
>
> Great!
>
> markusN
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20171205/18c170db/attachment.html>