[GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Glynn Clements]

Dylan Beaudette wrote:

> > If it's a pthread issue, using "export WORKERS=0" from the shell
> > before running t.rast.mapcalc should make it go away.
> 
> I don't think that it has anything to do with pthreads... Does the
> "--without-pthread" configure argument actually work?

--without-pthread is the default. You can check whether GRASS was
built with pthread support by searching for HAVE_PTHREAD_H in
include/grass/config.h. The configure script only checks for
<pthread.h> (and defines that macro) if --with-pthread is given.

> > r.mapcalc uses a mutex for each map to prevent multiple threads
> > accessing a single map concurrently. Accessing different maps from
> > multiple threads should be safe ... for normal GRASS rasters, and
> > provided that no MASK raster is present (otherwise it ends up
> > accessing the MASK raster from multiple threads concurrently, which
> > results in the "Error reading raster data ..." error, due to a race
> > condition between the lseek() and the read()).
> 
> Got it. This is valuable information. Is it documented anywhere other
> than GRASS-user and GRASS-dev mailing lists? How hard would it be to
> issue a warning when someone tries parallel access to the MASK? It
> would seem that some of the new t.* modules could be affected by this
> reality as many of them support parallel processes. Is it the duty of
> the module author to warn of such pitfalls?

The MASK issue should have been fixed with r65591 (in trunk, not sure
if it has been backported to releasebranch_7_0 yet).

Ultimately, if a module uses pthreads itself, it has to ensure that
any functions which aren't thread-safe aren't called concurrently from
multiple threads.

Accessing different maps from multiple threads is "intended" to be
safe. But the issue with the MASK was overlooked, as was the potential
for issues with GDAL-linked rasters.

I'm starting to think that r.mapcalc's pthread support to may be more
trouble than it's worth. Unless the expression is particularly
complex, r.mapcalc tends to be I/O-bound, and pthreads doesn't really
help with that.

At a minimum, I may change the thread pool code (lib/gis/worker.c) so
that the default number of workers (if WORKERS isn't set) is zero,
i.e. multiple threads will only be used if specifically requested by
the user.

> > But for GDAL-linked (r.external) rasters, there may be re-entrancy
> > issues with GDAL itself, meaning that even accessing different maps
> > from multiple threads is unsafe.
> 
> This is troubling. Is there anyway to empirically determine what is
> safe? Last week worked on a project where I was processing daily
> rasters for a 30 year interval, involving 3 climatic variables. Due to
> the large number of files, I had to work with "external" data sources.
> For the most part, everything went as expected when the results of
> (parallel) r.mapcalc expressions were CELL maps. Not so when the
> results were FCELL or DCELL maps.

The main problem with GDAL is that it's essentially a collection of
distinct modules in a common framework. Unless they have (and enforce)
a policy that GDALRasterIO() must be re-entrant (at least for
different maps), it may be the case that concurrent access is safe for
some formats but not others.

> > Fixing that is potentially problematic. r.mapcalc doesn't know whether
> > its inputs are GDAL-linked (this feature is supposed to be transparent
> > to modules; there isn't a Rast_is_gdal(int fd) or similar function),
> > and lib/raster itself doesn't have any knowledge of pthreads (lib/gis
> > provides functions for managing a thread pool; r.mapcalc/r3.mapcalc
> > are currently the only modules using these functions, and r3.mapcalc
> > specifically sets WORKERS=0 because lib/raster3d isn't thread-safe).
> 
> OK. Does setting "WORKERS=0" do anything when GRASS is compiled
> without pthreads?

No.

If you're experiencing these problems with WORKERS=0 or with GRASS
compiled without pthread support, then the problem is something else
entirely.

-- 
Glynn Clements <glynn at gclements.plus.com>