[GRASS-dev] [GRASS GIS] #2764: corrupt data written to FCELL and DCELL rasters, hard to re-produce

[GRASS GIS]

#2764: corrupt data written to FCELL and DCELL rasters, hard to re-produce
---------------------+-------------------------
  Reporter:  dylan   |      Owner:  grass-dev@…
      Type:  defect  |     Status:  new
  Priority:  normal  |  Milestone:  7.2.3
 Component:  Raster  |    Version:  unspecified
Resolution:          |   Keywords:
       CPU:  x86-64  |   Platform:  Linux
---------------------+-------------------------

Comment (by dylan):

 Replying to [comment:32 mmetz]:

 > Markus Neteler in particular spent a lot of time to fix various systems
 for parallel execution of GRASS commands. GRASS itself was never the
 problem, instead the main problem was that the multiple outputs to be
 written to a single storage device were too much for that storage device.

 OK. Good to know. Are there any other diagnostics for these kind of
 problems, other than looking through the output from `dmesg` or kernel
 messages? I typically run `dstat` while developing parallel processing
 scripts, but I haven't noticed any disk-thrashing in this case. I see that
 the SSD is at about 5% of its "write bandwidth" as reported by `dstat`.

 Each instance of `r.sun` is writing out ~5000 rows of data over a period
 of about 8 minutes, so that is 5000 rows * 8 processes * 1/8 proc per
 minute * 1/60 minutes per second = ~80 write operations per second
 (assuming rows are written as processed). Are rows written 1-by-1, or in
 batches?


 > > I know this is a lot to ask, but did you try testing using ZLIB
 compression and running it multiple times? It took a couple of tiles
 before I noticed the error.
 >
 > I did use ZLIB compression when running the test with the data and
 scripts provided. Do you mean I should run the test several times with the
 same data?

 Yes (please). It wasn't until I had ran a couple of tiles that I
 encountered errors.

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2764#comment:33>
GRASS GIS <https://grass.osgeo.org>