[GRASS-dev] any special reason why choice of raster compression method is done via environmental variable ?

[Moritz Lennert]

Coming back to this to continue the discussion:

On 15/12/19 21:41, Markus Metz wrote:
> 
> 
> On Tue, Dec 10, 2019 at 6:47 AM Moritz Lennert 
> <mlennert at club.worldonline.be <mailto:mlennert at club.worldonline.be>> wrote:
>  >
>  > On 9/12/19 22:50, Markus Metz wrote:
>  > >   Hi Moritz,
>  > >
>  > > On Wed, Dec 4, 2019 at 10:28 AM Moritz Lennert
>  > > <mlennert at club.worldonline.be <mailto:mlennert at club.worldonline.be> 
> <mailto:mlennert at club.worldonline.be 
> <mailto:mlennert at club.worldonline.be>>> wrote:
>  > >  >
>  > >  > Hi Markus,
>  > >  >
>  > >  > In recent days, I've been confronted several times with the issue of
>  > >  > people trying to share data among themselves, but using different
>  > >  > versions of GRASS, and so raster data compressed in a more recent
>  > >  > version of GRASS was not usable in an older version of GRASS.
>  > >  >
>  > >  > Now, I agree that generally the solution is to tell people to 
> use the
>  > >  > latest and greatest, but this is not always possible / it is not
>  > >  > necessarily highest on the list of priorities of people to see 
> how they
>  > >  > can install the latest version of GRASS within their particular
>  > > environment.
>  > >  >
>  > >  > Obviously, those with the latest version of GRASS can simple 
> recompress
>  > >  > using ZLIB. However, compression method is defined as an environment
>  > >  > variable. This is somewhat daunting to many MS Windows users out 
> there.
>  > >  > Is there any specific reason that lead to the choice of not using a
>  > >  > parameter to allow the choice of compression method (possibly to
>  > >  > override a default that is still defined by an environment 
> variable) ?
>  > >
>  > > Such a parameter would need to be added to every module creating raster
>  > > output.
>  >
>  > My request is more linked to use cases where one would like to share
>  > data (e.g. with r.pack) with other GRASS GIS users who do not
>  > necessarily have access to the same compression method, not necessarily
>  > to changing the default compression method. I was just wondering whether
>  > it might be easily possible to just implement r.compress method= as a
>  > quick way to recompress a specific map with a chosen method, overriding
>  > the default method. Currently, to do that, you have to change the
>  > default method by changing the env variable, run r.compress, then change
>  > the variable back to the value one wishes generally to use as default.
> 
> recompressing an existing raster map seems like a waste of time. Why not 
> compress it in the first place with the appropriate method?

Because what you consider the appropriate method at the time creation of 
the data (and still the appropriate method for your actual work) might 
not be the appropriate method for sharing with someone using a different 
system.

> 
> Recompressing an existing map with a chosen method via r.compress 
> method=XY means that a new raster map should not be compressed with the 
> default method but another supported method. This would require a new 
> function in lib/raster like Rast_open_new_compression_method() that 
> overrides the default compression method. Of course this is possible, 
> but it requires substantial changes to lib/raster.

Ok, I understand that this is complicated.

> 
> About the use cases:
> People want to share a mapset: in this case all GRASS raster maps should 
> be compressed with the most compatible compression method (ZLIB). 
> Currently, this can be done with an env var. Alternatively, a new 
> mechanism via g.gisenv could be established.

This would already make life easier for Windows users who (for the 
majority) are not used to env variables (and this is becoming more and 
more true of GNU/Linux users as well...).

> Another option could be to set the default compression method per 
> mapset, to be stored in in the mapset's VAR file, to be set with e.g. 
> r.compress method=.

That would probably be a nice solution. One could then create an 
"Export" mapset which can then be easily shared.

>  > Obviously, you can always just export as tiff and share that, but that
>  > just feels less elegant. Anyway, this is probably somewhat of a luxury
>  > problem :-)
> 
> This only a luxury problem if you don't need to care about processing 
> speed and disk usage. The different compression methods by now available 
> in GRASS allow for optimization for speed (LZ4) or for optimization for 
> disk usage (BZIP2 for larger CELL maps). An easy (easier) way to set the 
> compression method seems reasonable.

I didn't mean that compression is a luxury problem, but rather that 
since the option exists of just exporting the map as a tiff, readable by 
most systems (and you can chose a sharable tiff compression method at 
export time), all the above is probably a bit less of a necessity, and 
I'm just making noise ;-)

Moritz