[GRASS-dev] NULL file compression: loop over uncompressed maps to save disk space

[Markus Neteler]

Hi,

while hunting for more GBs on my local disk I found many raster maps
with a still uncompressed NULL files (no surprise since the optional
new NULL compression was introduced in 7.2.0).

As an example - EU DEM25m:

uncompressed NULL file:
6000000000 Apr 13  2016 ./eu_laea/PERMANENT/cell_misc/eu_dem25/null

compressed NULL file:
32108798 Jan  3 15:09 eu_laea/PERMANENT/cell_misc/eu_dem25/nullcmpr

Ratio:
> 32108798 / 6000000000
[1] 0.005351466

... quite an improvement :-)

Having tons of raster maps here I thought of running r.null over all
raster maps. In general it is:

export GRASS_COMPRESS_NULLS=1
r.null -z myrastermap

Attached a patch which adds a second line of output to "r.compress -p
myrastermap" in order to check the actual compression state of a map:

r.compress -p eu_dem25
<eu_dem25> is compressed (method 2: ZLIB). Data type: FCELL
<eu_dem25> has an uncompressed NULL file

After compression it looks like this:

r.compress -p eu_dem25
<eu_dem25> is compressed (method 2: ZLIB). Data type: FCELL
<eu_dem25> has a compressed NULL file

Now, how to use that:

# all in one (check if NULL is compressed, if no, do it otherwise don't touch):
r.compress -p eu_dem25 2>&1 | grep uncompressed && r.null -z eu_dem25

Questions:
I believe that an additional -g flag for shell style printing would be
useful as well).
Maybe with a -g flag no need to use the stderr redirect?
Any better ideas here? (if yes, feel free to submit to SVN for testing)


################
Since r.null -z doesn't do anything useful if GRASS_COMPRESS_NULLS is
not set I have tried (!) to add a G_message() to tell the user if that
variable is set or not.
r.null -z eu_dem25
The GRASS_COMPRESS_NULLS environment variable is currently set
6%...

But it *always* tells that it is set, so my getenv() parsing is wrong.
Can anyone help please? Also attached...

thanks,
Markus
-------------- next part --------------
Index: raster/r.compress/main.c
===================================================================
--- raster/r.compress/main.c	(revision 70217)
+++ raster/r.compress/main.c	(working copy)
@@ -49,6 +49,9 @@
 static int pprint(char *);
 static int doit(char *, int, RASTER_MAP_TYPE);
 
+/* compressed null file, see lib/raster/open.c */
+#define NULLC_FILE "nullcmpr"
+
 int main(int argc, char *argv[])
 {
     int stat;
@@ -338,6 +341,12 @@
                          (map_type == FCELL_TYPE ? "FCELL" : "??"))));
     }
 
+    if (G_find_file2_misc("cell_misc", NULLC_FILE, name, G_mapset())) {
+        G_message(_("<%s> has a compressed NULL file"), name);
+    } else {
+        G_message(_("<%s> has an uncompressed NULL file"), name);
+    }
+
     return 0;
 }
 
Index: raster/r.null/main.c
===================================================================
--- raster/r.null/main.c	(revision 70217)
+++ raster/r.null/main.c	(working copy)
@@ -198,6 +198,11 @@
     if (recreate) {
 	int in_fd;
 	/* write a file of no-nulls */
+
+	char *nullcompr = getenv("GRASS_COMPRESS_NULLS");
+        int isnullcompr = (!nullcompr || atoi(nullcompr)) ? 1 : 0;
+	G_message(_("The GRASS_COMPRESS_NULLS environment variable is currently %s"), isnullcompr ? "set" : "unset");
+
 	null_bits = Rast__allocate_null_bits(cellhd.cols);
 	Rast__init_null_bits(null_bits, cellhd.cols);