[GRASS-dev] Error reading raster data for row xxx (only when using r.series and t.rast.series)

[Dylan Beaudette]

Hi Moritz,

Not using a MASK in this case. Fairly basic work-flow, using "tmin"
and "tmax" (Space Time Raster Dataset) loaded via r.external.


# make a where clause for finding the current year
  wc="strftime('%Y', start_time) = '"$year"'"

  # extract current dataset: "copies" by reference
  t.rast.extract --q --o input=tmin output=tmin_subset
basename=tmin_subset where="$wc"
  t.rast.extract --q --o input=tmax output=tmax_subset
basename=tmax_subset where="$wc"

  # GDD for current year: slow from external rasters
  # NOTE: 2 CPUs so that disk isn't thrashed
  t.rast.mapcalc --o nprocs=2 method=equal
input=tmin_subset,tmax_subset output=gdd basename=gdd expr="(((
min((tmax_subset * 1.8 + 32.0), 86.0) + max((tmin_subset * 1.8 +
32.0), 50) ) / 2) - 50)"


... the error occurs at this step in a non-deterministic pattern.
Strangely enough, the errors are more frequent during the morning
hours vs. afternoon hours!

  # sum GDD for this year: fast from SSD, but only single CPU thread
  t.rast.series --o --q in=gdd out=gdd_$year method=sum


Thanks,
Dylan



On Tue, Oct 13, 2015 at 1:04 AM, Moritz Lennert
<mlennert at club.worldonline.be> wrote:
> On 12/10/15 23:35, Dylan Beaudette wrote:
>>
>> Hi,
>>
>> Over the last couple of years I have noticed a very strange raster
>> corruption (?) issues when using r.series, and now more recently,
>> t.rast.series. Typically, I'll generate a large number of maps with
>> r.sun or t.rast.mapcalc and then aggregate the series with r.series or
>> t.rast.series. About 50% of the time the command runs as expected, the
>> other half of the time r.series or t.rast.series gives me an error
>> like this:
>>
>> Error reading raster data for row xxx (testmap)
>>
>> After this error, I can no longer perform any kind of operation on map
>> "testmap" without the dreaded Error reading raster data for row xxx...
>>
>> The situation was worse when using a MASK map, possibly related to a
>> similar (fixed?) issue discussed in this thread:
>>
>> http://lists.osgeo.org/pipermail/grass-dev/2015-July/075627.html
>>
>> Within that thread, Glynn mentioned that this type of error was
>> probably related to pthreads and concurrent processes. The temporary
>> fix entailed:
>>
>> export WORKERS=0
>>
>> I have tried this on my machine but the results are the same,
>> non-deterministic corruption (?) of one input to r.series or
>> t.rast.series.
>>
>> I have encountered this error on several disks, mirrored HDDs, single
>> HDD, and now on an SSD. I don't think that this is a disk problem,
>> rather, something that r.series or t.rast.series is "doing" to the
>> files it operates on.
>>
>> Is there some possibility that one of these commands is leaving a file
>> "open" or in some kind of intermediate state that prevents subsequent
>> commands from accessing the file?
>>
>> I'll try to create a sample dataset to send over. In the meantime is
>> there any kind of diagnostic information that I can report back with?
>
>
> Are you using a mask, as was the case in the thread you cite ?
>
> Moritz