[GRASS-dev] NULL file compression: loop over uncompressed maps to save disk space

[Markus Metz]

On Tue, Jan 3, 2017 at 6:08 PM, Markus Neteler <neteler at osgeo.org> wrote:
>
> Hi,
>
> while hunting for more GBs on my local disk I found many raster maps
> with a still uncompressed NULL files (no surprise since the optional
> new NULL compression was introduced in 7.2.0).
>
> As an example - EU DEM25m:
>
> uncompressed NULL file:
> 6000000000 Apr 13  2016 ./eu_laea/PERMANENT/cell_misc/eu_dem25/null
>
> compressed NULL file:
> 32108798 Jan  3 15:09 eu_laea/PERMANENT/cell_misc/eu_dem25/nullcmpr
>
> Ratio:
> > 32108798 / 6000000000
> [1] 0.005351466
>
> ... quite an improvement :-)
>
> Having tons of raster maps here I thought of running r.null over all
> raster maps. In general it is:
>
> export GRASS_COMPRESS_NULLS=1
> r.null -z myrastermap
>
> Attached a patch which adds a second line of output to "r.compress -p
> myrastermap" in order to check the actual compression state of a map:
>
> r.compress -p eu_dem25
> <eu_dem25> is compressed (method 2: ZLIB). Data type: FCELL
> <eu_dem25> has an uncompressed NULL file
>
> After compression it looks like this:
>
> r.compress -p eu_dem25
> <eu_dem25> is compressed (method 2: ZLIB). Data type: FCELL
> <eu_dem25> has a compressed NULL file
>
> Now, how to use that:
>
> # all in one (check if NULL is compressed, if no, do it otherwise don't
touch):
> r.compress -p eu_dem25 2>&1 | grep uncompressed && r.null -z eu_dem25
>
> Questions:
> I believe that an additional -g flag for shell style printing would be
> useful as well).
> Maybe with a -g flag no need to use the stderr redirect?
> Any better ideas here? (if yes, feel free to submit to SVN for testing)

I have added your changes and a new shell style option to trunk in r70337.
Note that this is not standard shell style because the module accepts
several input maps, thus compression info is written to stdout as one line
per input map. The format is
input map name|data type|name of data compression method|NULL file
compression
e.g.
eu_dem25|FCELL|ZLIB|NO
or
eu_dem25|FCELL|BZIP2|YES

>
>
> ################
> Since r.null -z doesn't do anything useful if GRASS_COMPRESS_NULLS is
> not set I have tried (!) to add a G_message() to tell the user if that
> variable is set or not.
> r.null -z eu_dem25
> The GRASS_COMPRESS_NULLS environment variable is currently set
> 6%...
>
> But it *always* tells that it is set, so my getenv() parsing is wrong.
> Can anyone help please? Also attached...

Try trunk r70338. This also fixes removal of a compressed NULL file: the
file name is nullcmpr, not null2

Markus M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-dev/attachments/20170111/af196c67/attachment.html>