[GRASS-dev] Testing the new ZSTD raster compression - was: Re: GRASS GIS raster files: LZW compression?

[Markus Neteler]

Hi,

(subject renamed for clarity, original thread:
part 1:
https://lists.osgeo.org/pipermail/grass-dev/2017-October/thread.html#86395
part 2:
https://lists.osgeo.org/pipermail/grass-dev/2017-December/086738.html
)

Here the benchmarking with SRTM 30m global with 5.4 billion pixels (run on
SSD disk):


Welcome to GRASS GIS 7.5.svn (r71892)
GRASS 7.5.svn (latlong):~ >

export GRASS_COMPRESSOR=ZSTD
time -p r.in.gdal /vsicurl/
https://www.datenatlas.de/geodata/public/srtmgl1/srtmgl1.003.tif
output=srtmgl1_v003_30m_ZSTD memory=3000
360 degree EW extent is exceeded by 0.0395247 cells
360 degree EW extent is exceeded by 1 cells
Importing raster map <srtmgl1_v003_30m_ZSTD>...
 100%
real 17697.11
user 17149.13
sys 195.68

g.region raster=srtmgl1_v003_30m_ZSTD -p
360 degree EW extent is exceeded by 0.0395247 cells
360 degree EW extent is exceeded by 1 cells
360 degree EW extent is exceeded by 1 cells
360 degree EW extent is exceeded by 1 cells
projection: 3 (Latitude-Longitude)
zone:       0
datum:      wgs84
ellipsoid:  wgs84
north:      60:00:00.5N
south:      56:00:00.5S
west:       180:00:00.5W
east:       180:00:00.5E
nsres:      0:00:01
ewres:      0:00:01
rows:       417601
cols:       1296001
cells:      541211313601
360 degree EW extent is exceeded by 1 cells

r.colors srtmgl1_v003_30m_ZSTD color=srtm_plus
Color table for raster map <srtmgl1_v003_30m_ZSTD> set to 'srtm_plus'

r.compress -p srtmgl1_v003_30m_ZSTD
360 degree EW extent is exceeded by 1 cells
<srtmgl1_v003_30m_ZSTD> is compressed (method 5: ZSTD). Data type: CELL
<srtmgl1_v003_30m_ZSTD> has a compressed NULL file



File size comparison:

# ZLIB compressed, human readable:
mundialis:~/grassdata/latlong/srtmgl1_30m$ find . -name srtmgl1_v003_30m |
sort | xargs du -h
4,0K    ./cats/srtmgl1_v003_30m
4,0K    ./cellhd/srtmgl1_v003_30m
293M    ./cell_misc/srtmgl1_v003_30m
157G    ./cell/srtmgl1_v003_30m
4,0K    ./colr/srtmgl1_v003_30m
4,0K    ./hist/srtmgl1_v003_30m

# ZLIB compressed, kB:
mundialis:~/grassdata/latlong/srtmgl1_30m$ find . -name srtmgl1_v003_30m |
sort | xargs du -k
4       ./cats/srtmgl1_v003_30m
4       ./cellhd/srtmgl1_v003_30m
299696  ./cell_misc/srtmgl1_v003_30m
163988608       ./cell/srtmgl1_v003_30m
4       ./colr/srtmgl1_v003_30m
4       ./hist/srtmgl1_v003_30m

ls -la cell_misc/srtmgl1_v003_30m/
total 299696
-rw-rw-r-- 1 mundialis mundialis 306883360 Okt 25 21:19 nullcmpr
-rw-rw-r-- 1 mundialis mundialis        13 Okt 25 21:19 range


# ZSTD compressed, kB:
mundialis:/scratch/grassdata/latlong/srtmgl1_30m$ find . -name
srtmgl1_v003_30m_ZSTD | sort | xargs du -k
4       ./cats/srtmgl1_v003_30m_ZSTD
4       ./cellhd/srtmgl1_v003_30m_ZSTD
299708  ./cell_misc/srtmgl1_v003_30m_ZSTD
145278592       ./cell/srtmgl1_v003_30m_ZSTD
4       ./colr/srtmgl1_v003_30m_ZSTD
4       ./hist/srtmgl1_v003_30m_ZSTD

ls -la cell_misc/srtmgl1_v003_30m_ZSTD/
total 299704
-rw-rw-r-- 1 mundialis mundialis 306883360 Dez  5 21:40 nullcmpr
-rw-rw-r-- 1 mundialis mundialis        22 Dez  5 21:40 stats
-rw-rw-r-- 1 mundialis mundialis        13 Dez  5 21:40 range

Ratio:
# CELL file
> 145278592 / 163988608
[1] 0.8859066

# null file
identical

Curiosity: would it makes sense to compress the "nullcmpr" file according
to the selected compression? Probably Markus M had already explained it but
I don't remember...

Best,
markusN

-- 
Markus Neteler, PhD
http://www.mundialis.de - free data with free software
http://grass.osgeo.org
http://courses.neteler.org/blog
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20171206/06d314fd/attachment.html>