[GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed
--------------------------+---------------------------
  Reporter:  sprice       |      Owner:  grass-dev@…
      Type:  enhancement  |     Status:  new
  Priority:  normal       |  Milestone:  7.1.0
 Component:  Raster       |    Version:  svn-trunk
Resolution:               |   Keywords:  ZLIB LZ4 ZSTD
       CPU:  OSX/Intel    |   Platform:  MacOSX
--------------------------+---------------------------

Comment (by wenzeslaus):

 In the diff I see the following pattern (this if from
 [source:grass/trunk/lib/raster/get_row.c#L84 lib/raster/get_row.c],
 `read_data_fp_compressed()` function):

 {{{
 #!diff
 -    if ((size_t) G_zlib_read(fcb->data_fd, readamount, data_buf, bufsize)
 != bufsize)
 +    cmp = G_alloca(readamount);
 +
 +    if (read(fcb->data_fd, cmp, readamount) != readamount) {
 +        G_freea(cmp);
         G_fatal_error(_("Error reading raster data for row %d of <%s>"),
                       row, fcb->name);
  }

 +    if (cmp[0] == '0') // flate.c::G_ZLIB_COMPRESSED_NO
 +        // Uncompressed
 +        memcpy(data_buf, cmp+1, bufsize);
 +    else
 +        if (cmp[0] == 3 || cmp[0] == 4)
 +//          LZ4_decompress_safe((char *)(cmp+1), (char *)data_buf,
 +//                              readamount-1, bufsize);
 +        LZ4_decompress_fast((char *)(cmp+1), (char *)data_buf, bufsize);
 +    else if (cmp[0] == 5)
 +        ZSTD_decompress(data_buf, bufsize, cmp+1, readamount-1);
 +    else
 +        G_zlib_expand(cmp+1, readamount-1, data_buf, bufsize);
 +
 +    G_freea(cmp);
 +}
 }}}

 In words, `G_zlib_read()` function is replaced by a block of code. Similar
 applies to `G_zlib_write()` function and to certain extent to
 `G_zlib_expand()` and `zlib_compress()` as well.

 My question is if it wouldn't be more advantageous to create some wrapper
 which would take the all necessary inputs including compression type and
 do the necessary switches and format specific things.

 I suppose this would make easier to add the same compression options also
 to 3D raster library where `G_zlib_read()` and `G_zlib_write()` functions
 are already used ([source:grass/trunk/lib/raster3d/fpcompress.c#L718
 lib/raster3d/fpcompress.c]) or perhaps even to some other code such as
 [source:grass/trunk/lib/segment lib/segment] where no compression is used.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:13>
GRASS GIS <https://grass.osgeo.org>