[gdal-dev] Intermittent raster read/write error

[Jon Morris]

Quick update:

I haven't found the cause of this error (or a reliable repro case), but have found that changing how often we call ds.FlushCache() makes it go away.

After a previous bug fix*, we changed to flushing the cache after every tile. Reverting it back to a longer interval (every 1e6 tiles) meant that we didn't see this write error.


* Changing the flush interval was due to another error though - if you mistakenly open the output dataset as read-only, there is no indication until the process is finished, or you call FlushCache, whichever is sooner. Is there a better way to determine that a write has failed? We've enabled gdal.UseExceptions, but if you call rb.WriteArray() on a read-only dataset, no error is raised. There is a write to stderr, but no exception.

Jon

From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> On Behalf Of Jon Morris
Sent: 07 March 2019 11:39
To: gdal-dev at lists.osgeo.org
Subject: [gdal-dev] Intermittent raster read/write error

Hello all,

I've had an intermittent error with reading and writing rasters and wondered if anyone else had seen something similar.

Tl;dr:
When writing tiles of data to a raster, sometimes one write will cause subsequent reads of that region to fail. This includes when reading the whole raster at once with ReadAsArray().

Background:
We have a python toolkit that uses GDAL functions on a range of GIS data, supported on both Windows and Linux. Raster processing is tiled, usually in blocks of 256 pixels. Our raster merge tool usually opens the files as a vrt then processes one block at a time. This is really inefficient when merging hundreds of files at once, as it requires a multi-dimensional numpy array. For this scenario, we calculate the total extent of the input rasters, create a blank output raster, then merge the input files with this raster one-by-one.

The problem:
Sometimes the write operation fails in some way, so that the next read fails. I've tried reading one of these bad rasters and it turned out there was just one bad tile, i.e. ds.ReadAsArray(xoff,yoff,xsize,ysize) works if you request some data that does not include the bad region. When I wrote nodata values to this one tile, the whole raster could be read again.

I've seen this happen for different datasets on different platforms but it is hard to reproduce the error exactly. If you leave out one of the input files, or change the output extent slightly, the problem does not occur. I'm suspecting it's a floating point rounding error, but can't really tell.

I'm still trying to track down the cause really, but I just wondered if this sounded familiar to anyone and/or if there were any solutions.

Thanks,

Jon


Jon Morris
Software Developer

Skype<sip:jon.morris at jbarisk.com>


T +44 (0) 1756 799919
www.jbarisk.com<http://www.jbarisk.com>

[Visit our website]<http://www.jbarisk.com> [http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png] <https://www.linkedin.com/in/jon-morris-a2897b4/>  [Follow us on Twitter] <https://twitter.com/jbarisk>

Find out more about us here: www.jbarisk.com<http://www.jbarisk.com/> and follow us on Twitter @JBARisk<http://twitter.com/JBARisk> and LinkedIn<https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT>

The JBA Group supports the JBA Trust.

All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.


JBA Risk Management Limited is registered in England, company number 07732946, South Barn, Broughton Hall, Skipton, North Yorkshire, BD23 3AE, Telephone: +441756799919


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20190314/80d54737/attachment.html>