[GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Markus Neteler]

On Thu, Oct 22, 2015 at 3:14 AM, Glynn Clements
<glynn at gclements.plus.com> wrote:
> Dylan Beaudette wrote:
...
> The MASK issue should have been fixed with r65591 (in trunk, not sure
> if it has been backported to releasebranch_7_0 yet).

Yes, in r65764 about 4 months ago.

> Ultimately, if a module uses pthreads itself, it has to ensure that
> any functions which aren't thread-safe aren't called concurrently from
> multiple threads.
>
> Accessing different maps from multiple threads is "intended" to be
> safe. But the issue with the MASK was overlooked, as was the potential
> for issues with GDAL-linked rasters.
>
> I'm starting to think that r.mapcalc's pthread support to may be more
> trouble than it's worth. Unless the expression is particularly
> complex, r.mapcalc tends to be I/O-bound, and pthreads doesn't really
> help with that.

Given all the related updated in GCC etc., perhaps it is possible to
implement reasonable openMP support now?

> At a minimum, I may change the thread pool code (lib/gis/worker.c) so
> that the default number of workers (if WORKERS isn't set) is zero,
> i.e. multiple threads will only be used if specifically requested by
> the user.

Maybe better since this question came up a few times.

>> > But for GDAL-linked (r.external) rasters, there may be re-entrancy
>> > issues with GDAL itself, meaning that even accessing different maps
>> > from multiple threads is unsafe.
>>
>> This is troubling. Is there anyway to empirically determine what is
>> safe? Last week worked on a project where I was processing daily
>> rasters for a 30 year interval, involving 3 climatic variables. Due to
>> the large number of files, I had to work with "external" data sources.
>> For the most part, everything went as expected when the results of
>> (parallel) r.mapcalc expressions were CELL maps. Not so when the
>> results were FCELL or DCELL maps.
>
> The main problem with GDAL is that it's essentially a collection of
> distinct modules in a common framework. Unless they have (and enforce)
> a policy that GDALRasterIO() must be re-entrant (at least for
> different maps), it may be the case that concurrent access is safe for
> some formats but not others.
>
>> > Fixing that is potentially problematic. r.mapcalc doesn't know whether
>> > its inputs are GDAL-linked (this feature is supposed to be transparent
>> > to modules; there isn't a Rast_is_gdal(int fd) or similar function),
>> > and lib/raster itself doesn't have any knowledge of pthreads (lib/gis
>> > provides functions for managing a thread pool; r.mapcalc/r3.mapcalc
>> > are currently the only modules using these functions, and r3.mapcalc
>> > specifically sets WORKERS=0 because lib/raster3d isn't thread-safe).
>>
>> OK. Does setting "WORKERS=0" do anything when GRASS is compiled
>> without pthreads?
>
> No.
>
> If you're experiencing these problems with WORKERS=0 or with GRASS
> compiled without pthread support, then the problem is something else
> entirely.

(apparently no such issue)


Markus


> --
> Glynn Clements <glynn at gclements.plus.com>
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev