[gdal-dev] Intermittent raster read/write error

[Jon Morris]

Hello all,

I've had an intermittent error with reading and writing rasters and wondered if anyone else had seen something similar.

Tl;dr:
When writing tiles of data to a raster, sometimes one write will cause subsequent reads of that region to fail. This includes when reading the whole raster at once with ReadAsArray().

Background:
We have a python toolkit that uses GDAL functions on a range of GIS data, supported on both Windows and Linux. Raster processing is tiled, usually in blocks of 256 pixels. Our raster merge tool usually opens the files as a vrt then processes one block at a time. This is really inefficient when merging hundreds of files at once, as it requires a multi-dimensional numpy array. For this scenario, we calculate the total extent of the input rasters, create a blank output raster, then merge the input files with this raster one-by-one.

The problem:
Sometimes the write operation fails in some way, so that the next read fails. I've tried reading one of these bad rasters and it turned out there was just one bad tile, i.e. ds.ReadAsArray(xoff,yoff,xsize,ysize) works if you request some data that does not include the bad region. When I wrote nodata values to this one tile, the whole raster could be read again.

I've seen this happen for different datasets on different platforms but it is hard to reproduce the error exactly. If you leave out one of the input files, or change the output extent slightly, the problem does not occur. I'm suspecting it's a floating point rounding error, but can't really tell.

I'm still trying to track down the cause really, but I just wondered if this sounded familiar to anyone and/or if there were any solutions.

Thanks,

Jon


Jon Morris
Software Developer

Skype<sip:jon.morris at jbarisk.com>


T +44 (0) 1756 799919
www.jbarisk.com<http://www.jbarisk.com>

[Visit our website]<http://www.jbarisk.com> [http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png] <https://www.linkedin.com/in/jon-morris-a2897b4/>  [Follow us on Twitter] <https://twitter.com/jbarisk>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20190307/855f4814/attachment-0001.html>