[GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter:  sprice       |      Owner:  grass-dev@…
      Type:  enhancement  |     Status:  new
  Priority:  normal       |  Milestone:  7.1.0
 Component:  Raster       |    Version:  svn-trunk
Resolution:               |   Keywords:  ZLIB LZ4 ZSTD
       CPU:  OSX/Intel    |   Platform:  MacOSX
--------------------------+---------------------------

Comment (by wenzeslaus):

 Replying to [comment:15 sprice]:
 > I considered writing a more general function for compression but I held
 off for two reasons:
 >
 > 1) There seems to be a lot of very specific code depending on exactly
 what you're doing.

 My point is to put these specific cases into one function. So for example,
 whatever in the comment:13, should be in one function. What I suggest
 should be simple to moving of the existing code unless I miss some
 important part.

 > For example, for read_data_fp_compressed() there was functionality
 hidden in G_zlib_read to save a header for each compressed row.

 Here I perhaps hit my limited understanding of the code in question. What
 is doing this job now? Or it is not needed?

 > Or the use of G_ZLIB_COMPRESSED_NO ('0') as a flag for FP compression,
 vs various integers defining CELL compression.

 This is the kind of things which I would like to have hidden in one place.
 (In this way, we can keep the strange things at one place and e.g. easily
 replace, literals by defines (which we should do anyway, BTW).

 > To keep backwards compatibility, I delt with the different formats (CELL
 vs NULL vs FCELL/DCELL) on a case-by-case basis.

 You mean e.g. `read_data_compressed()` versus `read_data_fp_compressed()`?
 I had no time to actually try to identify all the differences, but the
 idea would be that these functions should deal with the differences among
 CELL, NULL and FCELL/DCELL while the proposed "`G_compresslib_write()`"
 function would deal with the differences between compression formats.

 > I'm concerned that as each edge case is considered (as mentioned above)
 there won't be all that much common code between the various compression
 uses.

 Another way to look at it is to think about, what do I have to replicate
 in 3D raster library where only `G_zlib_*` is used now. The things I need
 to replicate to enable other compressions should be in the
 "`G_compresslib_*()`" functions.

 I won't be able to give it attention it deserves in the following weeks,
 so please don't expect any actual code from me now. But feel free to
 oppose me or try it yourself in the mean time.

--
Ticket URL: <http://trac.osgeo.org/grass/ticket/2750#comment:16>
GRASS GIS <https://grass.osgeo.org>