[GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter:  sprice       |      Owner:  grass-dev@…
      Type:  enhancement  |     Status:  new
  Priority:  normal       |  Milestone:  7.1.0
 Component:  Raster       |    Version:  svn-trunk
Resolution:               |   Keywords:  ZLIB LZ4 ZSTD
       CPU:  OSX/Intel    |   Platform:  MacOSX
--------------------------+---------------------------

Comment (by wenzeslaus):

 Replying to [comment:19 mmetz]:
 > I have implemented something like this recently and added support for
 LZ4 (and BZIP2) compression to my local copy of GRASS trunk.
 >
 > I am attaching a patch for trunk r66775 and an archive with new files to
 go to lib/gis

 The design in the patch looks really good. I did tests and benchmark but
 it was not as successful as I hoped for.

 The benchmark was the same as [comment:10 before] but modified for this
 patch. It is more for testing than benchmark anyway. It was on 30,000,000
 cells but perhaps the previous one was on more and it is not completely
 precise overall due to some other computations running at the same time
 (although the result is from 10 runs aggregated by ''perf'').

 || type || write || read ||
 || NONE || 2.58 || 0.72 ||
 || ZLIB || 1.52 || 0.93 ||
 || LZ4 || 1.56 || 0.85 ||

 RLE and BZIP2 are missing because I got and error when writing using
 r.mapcalc with RLE:

 {{{
 *** Error in `r.mapcalc': free(): invalid next size (normal):
 0x0000000000edacd0 ***
 r.mapcalc: Aborted
 }}}

 The the commands are:

 {{{
 export GRASS_COMPRESSOR=RLE
 r.mapcalc expression="test_rast_z_base = rand(double(-200.), 900)"
 seed=100
 r.mapcalc expression=test_rast_rle=double(test_rast_z_base)
 }}}

 I guess I'm missing something in configure because with BZIP2 it says
 "ERROR: GRASS needs to be compiled with BZIP2 for BZIP2 compression".

 The report from `r.compress`:

 {{{
 <test_rast_z_base> is compressed (level 2: DEFLATE). Data type: <DCELL>
 <test_rast_none> is uncompressed (level 0: NONE). Data type: <DCELL>
 <test_rast_zlib> is compressed (level 2: DEFLATE). Data type: <DCELL>
 <test_rast_lz4> is compressed (level 3: DEFLATE). Data type: <DCELL>
 }}}

 When running the tests with

 {{{
 grass71 /grassdata/ncarolina_smp_base/practice1 --exec python -m
 grass.gunittest.main--location ncarolina_smp_base --location-type nc
 }}}

 I get

 {{{
 ERROR: Error reading raster data for row 0 of <elevation>
 }}}

 from many raster-related tests. Sometimes it is a different row number.
 The environmental variable `GRASS_COMPRESSOR` was not set. I didn't get if
 you already meant it to be backwards compatible with existing maps.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:20>
GRASS GIS <https://grass.osgeo.org>