[GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
trac at osgeo.org
Tue Nov 10 00:30:29 PST 2015
#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
Reporter: sprice | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
CPU: OSX/Intel | Platform: MacOSX
Comment (by mmetz):
Replying to [comment:20 wenzeslaus]:
> Replying to [comment:19 mmetz]:
> > I have implemented something like this recently and added support for
LZ4 (and BZIP2) compression to my local copy of GRASS trunk.
> > I am attaching a patch for trunk r66775 and an archive with new files
to go to lib/gis
> The design in the patch looks really good. I did tests and benchmark but
it was not as successful as I hoped for.
> The benchmark was the same as [comment:10 before] but modified for this
patch. It is more for testing than benchmark anyway. It was on 30,000,000
I tested on 552,000,000 cells
but perhaps the previous one was on more and it is not completely precise
overall due to some other computations running at the same time (although
the result is from 10 runs aggregated by ''perf'').
> || type || write || read ||
> || NONE || 2.58 || 0.72 ||
> || ZLIB || 1.52 || 0.93 ||
> || LZ4 || 1.56 || 0.85 ||
> RLE and BZIP2 are missing because I got and error when writing using
r.mapcalc with RLE:
> *** Error in `r.mapcalc': free(): invalid next size (normal):
> r.mapcalc: Aborted
Fixed in the updated patch "compressors.patch"
> The the commands are:
> export GRASS_COMPRESSOR=RLE
> r.mapcalc expression="test_rast_z_base = rand(double(-200.), 900)"
> r.mapcalc expression=test_rast_rle=double(test_rast_z_base)
RLE compression is not and was never supported for fp maps. Also, if you
use r.mapcalc's rand() function, you test the rand() function and not the
compressor because all compressors are much faster than the rand()
function. (BTW, there are nice fast random number generators in gsl).
> I guess I'm missing something in configure because with BZIP2 it says
"ERROR: GRASS needs to be compiled with BZIP2 for BZIP2 compression".
You will need to hack in BZIP2 support, or make distclean, apply the patch
"bzipsupport.patch", configure --with-bzlib, make
> The report from `r.compress`:
I have not updated `r.compress`. For `r.compress`, and for meaningful
warning/error messages, something like `char *G_compressor_name(int
compressor)` that gives the compressor name could be useful.
> When running the tests with
> grass71 /grassdata/ncarolina_smp_base/practice1 --exec python -m
grass.gunittest.main--location ncarolina_smp_base --location-type nc
> I get
> ERROR: Error reading raster data for row 0 of <elevation>
> from many raster-related tests. Sometimes it is a different row number.
The environmental variable `GRASS_COMPRESSOR` was not set. I didn't get if
you already meant it to be backwards compatible with existing maps.
Fixed in the new patch "compressors.patch". I meant it to be backwards
compatible, but that one slipped through: old fp map compression is
sometimes 1, sometimes 2, but both 1 and 2 mean ZLIB.
Thanks for testing!
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:21>
GRASS GIS <https://grass.osgeo.org>
More information about the grass-dev