[GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter:  sprice       |      Owner:  grass-dev@…
      Type:  enhancement  |     Status:  new
  Priority:  normal       |  Milestone:  7.1.0
 Component:  Raster       |    Version:  svn-trunk
Resolution:               |   Keywords:  ZLIB LZ4 ZSTD
       CPU:  OSX/Intel    |   Platform:  MacOSX
--------------------------+---------------------------

Comment (by mmetz):

 Replying to [comment:20 wenzeslaus]:
 > Replying to [comment:19 mmetz]:
 > > I have implemented something like this recently and added support for
 LZ4 (and BZIP2) compression to my local copy of GRASS trunk.
 > >
 > > I am attaching a patch for trunk r66775 and an archive with new files
 to go to lib/gis
 >
 > The design in the patch looks really good. I did tests and benchmark but
 it was not as successful as I hoped for.
 >
 > The benchmark was the same as [comment:10 before] but modified for this
 patch. It is more for testing than benchmark anyway. It was on 30,000,000
 cells

 I tested on 552,000,000 cells

 but perhaps the previous one was on more and it is not completely precise
 overall due to some other computations running at the same time (although
 the result is from 10 runs aggregated by ''perf'').
 >
 > || type || write || read ||
 > || NONE || 2.58 || 0.72 ||
 > || ZLIB || 1.52 || 0.93 ||
 > || LZ4 || 1.56 || 0.85 ||
 >
 > RLE and BZIP2 are missing because I got and error when writing using
 r.mapcalc with RLE:
 >
 {{{
 > *** Error in `r.mapcalc': free(): invalid next size (normal):
 0x0000000000edacd0 ***
 > r.mapcalc: Aborted
 }}}

 Fixed in the updated patch "compressors.patch"
 >
 > The the commands are:
 >
 > {{{
 > export GRASS_COMPRESSOR=RLE
 > r.mapcalc expression="test_rast_z_base = rand(double(-200.), 900)"
 seed=100
 > r.mapcalc expression=test_rast_rle=double(test_rast_z_base)
 > }}}

 RLE compression is not and was never supported for fp maps. Also, if you
 use r.mapcalc's rand() function, you test the rand() function and not the
 compressor because all compressors are much faster than the rand()
 function. (BTW, there are nice fast random number generators in gsl).

 >
 > I guess I'm missing something in configure because with BZIP2 it says
 "ERROR: GRASS needs to be compiled with BZIP2 for BZIP2 compression".

 You will need to hack in BZIP2 support, or make distclean, apply the patch
 "bzipsupport.patch", configure --with-bzlib, make
 >
 > The report from `r.compress`:
 I have not updated `r.compress`. For `r.compress`, and for meaningful
 warning/error messages, something like `char *G_compressor_name(int
 compressor)` that gives the compressor name could be useful.
 >
 > When running the tests with
 >
 > {{{
 > grass71 /grassdata/ncarolina_smp_base/practice1 --exec python -m
 grass.gunittest.main--location ncarolina_smp_base --location-type nc
 > }}}
 >
 > I get
 >
 {{{
 > ERROR: Error reading raster data for row 0 of <elevation>
 }}}
 >
 > from many raster-related tests. Sometimes it is a different row number.
 The environmental variable `GRASS_COMPRESSOR` was not set. I didn't get if
 you already meant it to be backwards compatible with existing maps.

 Fixed in the new patch "compressors.patch". I meant it to be backwards
 compatible, but that one slipped through: old fp map compression is
 sometimes 1, sometimes 2, but both 1 and 2 mean ZLIB.

 Thanks for testing!

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:21>
GRASS GIS <https://grass.osgeo.org>