[GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
GRASS GIS
trac at osgeo.org
Mon Nov 9 18:10:25 PST 2015
#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
Reporter: sprice | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------
Comment (by wenzeslaus):
Replying to [comment:19 mmetz]:
> I have implemented something like this recently and added support for
LZ4 (and BZIP2) compression to my local copy of GRASS trunk.
>
> I am attaching a patch for trunk r66775 and an archive with new files to
go to lib/gis
The design in the patch looks really good. I did tests and benchmark but
it was not as successful as I hoped for.
The benchmark was the same as [comment:10 before] but modified for this
patch. It is more for testing than benchmark anyway. It was on 30,000,000
cells but perhaps the previous one was on more and it is not completely
precise overall due to some other computations running at the same time
(although the result is from 10 runs aggregated by ''perf'').
|| type || write || read ||
|| NONE || 2.58 || 0.72 ||
|| ZLIB || 1.52 || 0.93 ||
|| LZ4 || 1.56 || 0.85 ||
RLE and BZIP2 are missing because I got and error when writing using
r.mapcalc with RLE:
{{{
*** Error in `r.mapcalc': free(): invalid next size (normal):
0x0000000000edacd0 ***
r.mapcalc: Aborted
}}}
The the commands are:
{{{
export GRASS_COMPRESSOR=RLE
r.mapcalc expression="test_rast_z_base = rand(double(-200.), 900)"
seed=100
r.mapcalc expression=test_rast_rle=double(test_rast_z_base)
}}}
I guess I'm missing something in configure because with BZIP2 it says
"ERROR: GRASS needs to be compiled with BZIP2 for BZIP2 compression".
The report from `r.compress`:
{{{
<test_rast_z_base> is compressed (level 2: DEFLATE). Data type: <DCELL>
<test_rast_none> is uncompressed (level 0: NONE). Data type: <DCELL>
<test_rast_zlib> is compressed (level 2: DEFLATE). Data type: <DCELL>
<test_rast_lz4> is compressed (level 3: DEFLATE). Data type: <DCELL>
}}}
When running the tests with
{{{
grass71 /grassdata/ncarolina_smp_base/practice1 --exec python -m
grass.gunittest.main--location ncarolina_smp_base --location-type nc
}}}
I get
{{{
ERROR: Error reading raster data for row 0 of <elevation>
}}}
from many raster-related tests. Sometimes it is a different row number.
The environmental variable `GRASS_COMPRESSOR` was not set. I didn't get if
you already meant it to be backwards compatible with existing maps.
--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:20>
GRASS GIS <https://grass.osgeo.org>
More information about the grass-dev
mailing list