[GRASS-user] Multiple `r.proj` requests on the same raster map(s)

[Markus Metz]

On Sat, Feb 24, 2018 at 10:06 PM, Nikos Alexandris <nik at nikosalexandris.net>
wrote:
>
> * Markus Metz <markus.metz.giswork at gmail.com> [2018-02-24 21:39:40 +0100]:
>
>> On Sat, Feb 24, 2018 at 9:25 PM, Nikos Alexandris <
nik at nikosalexandris.net>
>> wrote:
>>>
>>>
>>> Dear community,
>>>
>>> I am asking for help to "debug" a situation.
>>>
>>> One ETRS89-based location, one Mapset with hundreds of land cover raster
>>> map tiles.
>>>
>>> Then, hundreds of UTM Zones-based Locations, subset of the WRS2 grid.
>>>
>>> Multiple `r.proj`es are running in parallel. However, only one at a time
>>> from inside one Mapset inside the respective UTM-Zone-based Location,
>>> requesting for one land cover raster map tile from the ETRS89-based
>>
>> Location.
>>>
>>>
>>> And, each `r.proj` is isolated running inside an independent docker
>>
>> container.
>>>
>>>
>>> To the question. Some tiles are cross-cut by more than one UTM-Zone.
>>> Hence, it happens that many UTM-Zones will/might request to read the
>>> same land cover raster map tile at the same time.
>>>
>>> That is one write per target Mapset/Location, yet highly probable
>>> concurrent read requests to the same raster map(s) in the source
>>> Mapset/Location.
>>>
>>> Is this bad?
>>
>>
>> Are you experiencing problems?
>
>
> Yes.
>
> (It's not easy for me to have access to the logs, as I
> don't directly have access to the scheduler. I got a copy though and I
> am reading through.)
>
> Looking at jobs logs, I read lots of ".gislock" lines.
> It might be some permission related issue. I partially operated directly
> (with my user-id) on many Locations.
>
> The operateor of the scheduler, has naturally, another user-id. I wonder
> if I should apply GRASS_SKIP_MAPSET_OWNER_CHECK=1 everywhere.

No, you need to run each process in a unique temporary mapset. Once you
have the final result, change the current mapset with g.mapset to the
common mapset where final results should stored and copy the final result
from the temporary mapset to the current mapset (the mapset to hold the
final results).

Alternatively/additionally, don't use the script grassXY to start a GRASS
session, instead define the GRASS environment with custom scripts (one for
the GRASS version to use, one for the database/location/mapset to use).
This avoids race conditions on a HPC system. A unique temporary mapset for
each process helps to avoid all sorts of concurrent access problems.

Markus M

>
>> I have been using concurrent read requests on the same raster maps on
>> different HPC systems for quite a few years by now and never got any
>> problems.
>
>
> Thank you Markusi. This is a very useful piece of information. It's
> reassuring that this part of the process is ok and the problem lies
elsewhere.
>
>> Concurrent write requests would be a different issue.
>
>
>> Markus M
>
>
> None here.
>
> Nikos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20180224/2085cb56/attachment.html>