[GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

On Mon, Oct 19, 2015 at 12:45 PM, Glynn Clements
<glynn at gclements.plus.com> wrote:
>
> Dylan Beaudette wrote:
>
>> I can now verify that t.rast.mapcalc is creating some raster maps with
>> corrupt (?) data. Corrupt in the sense that subsequent reading of the
>> maps results in the "Error reading raster data for row ..." error.
>> Just in case anyone is interested, I have opened a ticket for more
>> informative errors raised by lib/raster/get_row.c
>> (https://trac.osgeo.org/grass/ticket/2762).
>>
>> As previously reported, errors seem to occur about 50-60% of the time
>> and _do not_ appear to be related to the number of concurrent
>> t.rast.mapcalc instances.
>
> t.rast.mapcalc relies upon r.mapcalc.

OK, thanks for the clarification. I'll be more specific in future
tests / posts / tickets.

> If it's a pthread issue, using "export WORKERS=0" from the shell
> before running t.rast.mapcalc should make it go away.

I don't think that it has anything to do with pthreads... Does the
"--without-pthread" configure argument actually work?

> r.mapcalc uses a mutex for each map to prevent multiple threads
> accessing a single map concurrently. Accessing different maps from
> multiple threads should be safe ... for normal GRASS rasters, and
> provided that no MASK raster is present (otherwise it ends up
> accessing the MASK raster from multiple threads concurrently, which
> results in the "Error reading raster data ..." error, due to a race
> condition between the lseek() and the read()).

Got it. This is valuable information. Is it documented anywhere other
than GRASS-user and GRASS-dev mailing lists? How hard would it be to
issue a warning when someone tries parallel access to the MASK? It
would seem that some of the new t.* modules could be affected by this
reality as many of them support parallel processes. Is it the duty of
the module author to warn of such pitfalls?

Again, not sure how to implement but happy to test / document.


> But for GDAL-linked (r.external) rasters, there may be re-entrancy
> issues with GDAL itself, meaning that even accessing different maps
> from multiple threads is unsafe.

This is troubling. Is there anyway to empirically determine what is
safe? Last week worked on a project where I was processing daily
rasters for a 30 year interval, involving 3 climatic variables. Due to
the large number of files, I had to work with "external" data sources.
For the most part, everything went as expected when the results of
(parallel) r.mapcalc expressions were CELL maps. Not so when the
results were FCELL or DCELL maps.


> Fixing that is potentially problematic. r.mapcalc doesn't know whether
> its inputs are GDAL-linked (this feature is supposed to be transparent
> to modules; there isn't a Rast_is_gdal(int fd) or similar function),
> and lib/raster itself doesn't have any knowledge of pthreads (lib/gis
> provides functions for managing a thread pool; r.mapcalc/r3.mapcalc
> are currently the only modules using these functions, and r3.mapcalc
> specifically sets WORKERS=0 because lib/raster3d isn't thread-safe).

OK. Does setting "WORKERS=0" do anything when GRASS is compiled
without pthreads?

Thank you,
Dylan


> --
> Glynn Clements <glynn at gclements.plus.com>