[GRASS-dev] NULL file compression: loop over uncompressed maps to save disk space

Markus Metz markus.metz.giswork at gmail.com
Wed Jan 11 01:34:24 PST 2017

On Tue, Jan 3, 2017 at 6:08 PM, Markus Neteler <neteler at osgeo.org> wrote:
> Hi,
> while hunting for more GBs on my local disk I found many raster maps
> with a still uncompressed NULL files (no surprise since the optional
> new NULL compression was introduced in 7.2.0).
> As an example - EU DEM25m:
> uncompressed NULL file:
> 6000000000 Apr 13  2016 ./eu_laea/PERMANENT/cell_misc/eu_dem25/null
> compressed NULL file:
> 32108798 Jan  3 15:09 eu_laea/PERMANENT/cell_misc/eu_dem25/nullcmpr
> Ratio:
> > 32108798 / 6000000000
> [1] 0.005351466
> ... quite an improvement :-)
> Having tons of raster maps here I thought of running r.null over all
> raster maps. In general it is:
> r.null -z myrastermap
> Attached a patch which adds a second line of output to "r.compress -p
> myrastermap" in order to check the actual compression state of a map:
> r.compress -p eu_dem25
> <eu_dem25> is compressed (method 2: ZLIB). Data type: FCELL
> <eu_dem25> has an uncompressed NULL file
> After compression it looks like this:
> r.compress -p eu_dem25
> <eu_dem25> is compressed (method 2: ZLIB). Data type: FCELL
> <eu_dem25> has a compressed NULL file
> Now, how to use that:
> # all in one (check if NULL is compressed, if no, do it otherwise don't
> r.compress -p eu_dem25 2>&1 | grep uncompressed && r.null -z eu_dem25
> Questions:
> I believe that an additional -g flag for shell style printing would be
> useful as well).
> Maybe with a -g flag no need to use the stderr redirect?
> Any better ideas here? (if yes, feel free to submit to SVN for testing)

I have added your changes and a new shell style option to trunk in r70337.
Note that this is not standard shell style because the module accepts
several input maps, thus compression info is written to stdout as one line
per input map. The format is
input map name|data type|name of data compression method|NULL file

> ################
> Since r.null -z doesn't do anything useful if GRASS_COMPRESS_NULLS is
> not set I have tried (!) to add a G_message() to tell the user if that
> variable is set or not.
> r.null -z eu_dem25
> The GRASS_COMPRESS_NULLS environment variable is currently set
> 6%...
> But it *always* tells that it is set, so my getenv() parsing is wrong.
> Can anyone help please? Also attached...

Try trunk r70338. This also fixes removal of a compressed NULL file: the
file name is nullcmpr, not null2

Markus M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20170111/af196c67/attachment.html>

More information about the grass-dev mailing list