[GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter:  sprice       |      Owner:  grass-dev@…
      Type:  enhancement  |     Status:  new
  Priority:  normal       |  Milestone:  7.1.0
 Component:  Raster       |    Version:  svn-trunk
Resolution:               |   Keywords:  ZLIB LZ4 ZSTD
       CPU:  OSX/Intel    |   Platform:  MacOSX
--------------------------+---------------------------

Comment (by neteler):

 Some MODIS LST benchmark, using a European LST coverage from
 http://gis.cri.fmach.it/eurolst/

 '''File size tests''': trunk, r67403 (build_date=2015-12-28)

 {{{
 # dataset
 r.univar -g lst_2002_196_average
 n=188341794
 null_cells=226948851
 cells=415290645
 min=-481
 max=4570
 ...


 ########################
 #### RLE
 export GRASS_COMPRESSOR=RLE
 export GRASS_COMPRESS_NULLS=1

 ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
 -rw-r--r-- 1 neteler gis       10 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis 51923025 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/null
 -rw-r--r-- 1 neteler gis       44 Sep 22 15:15
 modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
 -rw-r--r-- 1 neteler gis       10 Dec 28 15:51
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis  1423931 Dec 28 15:51
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

 ls -la modis_lst_daily*/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Jun  9  2014
 modis_lst_daily/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 529008029 Dec 28 15:51
 modis_lst_daily_compr/cell/lst_2002_196_average


 ########################
 #### BZIP Level 1
 export GRASS_COMPRESSOR=ZLIB
 export GRASS_ZLIB_LEVEL=1
 export GRASS_COMPRESS_NULLS=1

 ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
 -rw-r--r-- 1 neteler gis       10 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis 51923025 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/null
 -rw-r--r-- 1 neteler gis       44 Sep 22 15:15
 modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
 -rw-r--r-- 1 neteler gis       10 Dec 28 16:49
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis  1579674 Dec 28 16:49
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

 ls -la modis_lst_daily*/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Jun  9  2014
 modis_lst_daily/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 263256860 Dec 28 16:49
 modis_lst_daily_compr/cell/lst_2002_196_average


 ########################
 #### BZIP Level 6 (default)
 export GRASS_COMPRESSOR=ZLIB
 export GRASS_ZLIB_LEVEL=6
 export GRASS_COMPRESS_NULLS=1

 ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
 -rw-r--r-- 1 neteler gis       10 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis 51923025 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/null
 -rw-r--r-- 1 neteler gis       44 Sep 22 15:15
 modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
 -rw-r--r-- 1 neteler gis       10 Dec 28 17:54
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis  1358116 Dec 28 17:54
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

 ls -la modis_lst_daily*/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Jun  9  2014
 modis_lst_daily/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Dec 28 17:54
 modis_lst_daily_compr/cell/lst_2002_196_average


 ########################
 #### LZ4
 export GRASS_COMPRESSOR=LZ4
 export GRASS_COMPRESS_NULLS=1

 ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
 -rw-r--r-- 1 neteler gis       10 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis 51923025 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/null
 -rw-r--r-- 1 neteler gis       44 Sep 22 15:15
 modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
 -rw-r--r-- 1 neteler gis       10 Dec 28 17:52
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis  1573603 Dec 28 17:52
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

 ls -la modis_lst_daily*/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Jun  9  2014
 modis_lst_daily/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 362932376 Dec 28 17:52
 modis_lst_daily_compr/cell/lst_2002_196_average

 ########################
 #### BZIP2
 export GRASS_COMPRESSOR=BZIP2
 export GRASS_COMPRESS_NULLS=1

 ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
 -rw-r--r-- 1 neteler gis       10 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis 51923025 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/null
 -rw-r--r-- 1 neteler gis       44 Sep 22 15:15
 modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
 -rw-r--r-- 1 neteler gis       10 Dec 28 17:49
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis  1824531 Dec 28 17:49
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

 ls -la modis_lst_daily*/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Jun  9  2014
 modis_lst_daily/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 180209506 Dec 28 17:49
 modis_lst_daily_compr/cell/lst_2002_196_average
 }}}

 Overall, ZLIB at Level 1 performs well followed by LZ4 (BZIP2 compresses
 best but was very slow; ZLIB level 6's small size gain over ZLIB level 1
 is "paid" by performance).

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:41>
GRASS GIS <https://grass.osgeo.org>