[GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
GRASS GIS
trac at osgeo.org
Sun Sep 27 20:28:33 PDT 2015
#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
Reporter: sprice | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------
Comment (by sprice):
I've gone ahead and added code to also compress FCELL/DCELL. Same tests at
before. Late 2013 Mac Pro, v10.10.5, 64 GB RAM, 3.5 GHz 6-Core. The
`r.mapcalc` commands are intended to test real-world write performance.
The `ls -l` command shows file sizes. `r.univar` is intended to show read
performance.
tl;dr: Write speed with LZ4 & ZSTD continues to be double the current
method. Read speed also significantly faster. LZ4 shows the best
performance if you have the disk space. ZSTD is very good all around, and
doesn't fill the hard drive as fast.
{{{
> time r.mapcalc expression="out_fp_orig=float(out_test_lz4hc)"
--overwrite
100%
real 4m57.518s
user 4m48.823s
sys 0m7.118s
> time r.mapcalc expression="out_fp_orig=float(out_test_lz4hc)"
--overwrite
100%
real 4m57.555s
user 4m49.946s
sys 0m6.382s
> export GRASS_INT_LZ4=1
> time r.mapcalc expression="out_fp_lz4=float(out_test_lz4hc)" --overwrite
100%
real 1m51.136s
user 1m44.766s
sys 0m5.879s
> time r.mapcalc expression="out_fp_lz4=float(out_test_lz4hc)" --overwrite
100%
real 1m51.898s
user 1m44.991s
sys 0m5.928s
> unset GRASS_INT_LZ4
> export GRASS_INT_LZ4HC=1
> time r.mapcalc expression="out_fp_lz4hc=float(out_test_lz4hc)"
--overwrite
100%
real 4m22.887s
user 4m15.556s
sys 0m6.206s
> time r.mapcalc expression="out_fp_lz4hc=float(out_test_lz4hc)"
--overwrite
100%
real 4m22.143s
user 4m15.164s
sys 0m6.075s
> unset GRASS_INT_LZ4HC
> export GRASS_INT_ZSTD=1
> time r.mapcalc expression="out_fp_zstd=float(out_test_lz4hc)"
--overwrite
100%
real 1m49.065s
user 1m43.401s
sys 0m5.183s
> time r.mapcalc expression="out_fp_zstd=float(out_test_lz4hc)"
--overwrite
100%
real 1m48.771s
user 1m43.238s
sys 0m5.093s
> ls -l vrt_test/PERMANENT/fcell/
total 46206984
-rw-r--r-- 1 sprice staff 7268345798 Sep 27 19:57 out_fp_lz4
-rw-r--r-- 1 sprice staff 5979129773 Sep 27 20:06 out_fp_lz4hc
-rw-r--r-- 1 sprice staff 4969771486 Sep 27 19:53 out_fp_orig
-rw-r--r-- 1 sprice staff 5440720320 Sep 27 20:10 out_fp_zstd
> unset GRASS_INT_ZSTD
> r.compress -p out_test_zlib
<out_test_zlib> is compressed (level 2: DEFLATE). Data type: <CELL>
> r.compress -p out_test_lz4
<out_test_lz4> is compressed (level 3: LZ4). Data type: <CELL>
> r.compress -p out_test_lz4hc
<out_test_lz4hc> is compressed (level 4: LZ4HC). Data type: <CELL>
> r.compress -p out_test_zstd
<out_test_zstd> is compressed (level 5: ZSTD). Data type: <CELL>
> r.compress -p out_fp_zstd
<out_fp_zstd> is compressed (level 5: ZSTD). Data type: <FCELL>
> r.compress -p out_fp_orig
<out_fp_orig> is compressed (level 2: DEFLATE). Data type: <FCELL>
> r.compress -p out_fp_lz4
<out_fp_lz4> is compressed (level 3: LZ4). Data type: <FCELL>
> r.compress -p out_fp_lz4hc
<out_fp_lz4hc> is compressed (level 4: LZ4HC). Data type: <FCELL>
> time r.univar out_fp_orig
100%
total null and non-null cells: 3526771952
total null cells: 1502448926
Of the non-null cells:
----------------------
n: 2024323026
minimum: 807
maximum: 32767
range: 31960
mean: 9385.79
mean of absolute values: 9385.79
standard deviation: 6620.52
variance: 4.38312e+07
variation coefficient: 70.5377 %
sum: 18999862195879
real 1m49.227s
user 1m46.152s
sys 0m2.843s
> time r.univar out_fp_lz4
100%
total null and non-null cells: 3526771952
total null cells: 1502448926
Of the non-null cells:
----------------------
n: 2024323026
minimum: 807
maximum: 32767
range: 31960
mean: 9385.79
mean of absolute values: 9385.79
standard deviation: 6620.52
variance: 4.38312e+07
variation coefficient: 70.5377 %
sum: 18999862195879
real 1m8.749s
user 1m4.596s
sys 0m3.564s
> time r.univar out_fp_lz4hc
100%
total null and non-null cells: 3526771952
total null cells: 1502448926
Of the non-null cells:
----------------------
n: 2024323026
minimum: 807
maximum: 32767
range: 31960
mean: 9385.79
mean of absolute values: 9385.79
standard deviation: 6620.52
variance: 4.38312e+07
variation coefficient: 70.5377 %
sum: 18999862195879
real 1m10.307s
user 1m6.546s
sys 0m3.352s
> time r.univar out_fp_zstd
100%
total null and non-null cells: 3526771952
total null cells: 1502448926
Of the non-null cells:
----------------------
n: 2024323026
minimum: 807
maximum: 32767
range: 31960
mean: 9385.79
mean of absolute values: 9385.79
standard deviation: 6620.52
variance: 4.38312e+07
variation coefficient: 70.5377 %
sum: 18999862195879
real 1m31.310s
user 1m27.916s
sys 0m3.044s
}}}
--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:1>
GRASS GIS <https://grass.osgeo.org>
More information about the grass-dev
mailing list