<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Arial",sans-serif;
color:black;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Hello all,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">I've had an intermittent error with reading and writing rasters and wondered if anyone else had seen something similar.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Tl;dr:
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">When writing tiles of data to a raster, sometimes one write will cause subsequent reads of that region to fail. This includes when reading the whole raster at once with ReadAsArray().<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Background:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">We have a python toolkit that uses GDAL functions on a range of GIS data, supported on both Windows and Linux. Raster processing is tiled, usually in blocks of 256 pixels. Our raster
merge tool usually opens the files as a vrt then processes one block at a time. This is really inefficient when merging hundreds of files at once, as it requires a multi-dimensional numpy array. For this scenario, we calculate the total extent of the input
rasters, create a blank output raster, then merge the input files with this raster one-by-one.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">The problem:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Sometimes the write operation fails in some way, so that the next read fails. I've tried reading one of these bad rasters and it turned out there was just one bad tile, i.e. ds.ReadAsArray(xoff,yoff,xsize,ysize)
works if you request some data that does not include the bad region. When I wrote nodata values to this one tile, the whole raster could be read again.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">I've seen this happen for different datasets on different platforms but it is hard to reproduce the error exactly. If you leave out one of the input files, or change the output extent
slightly, the problem does not occur. I'm suspecting it's a floating point rounding error, but can't really tell.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">I'm still trying to track down the cause really, but I just wondered if this sounded familiar to anyone and/or if there were any solutions.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Jon<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:black;mso-fareast-language:EN-GB">Jon Morris<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#F6A124;mso-fareast-language:EN-GB">Software Developer<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#F6A124;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:#F6A124;mso-fareast-language:EN-GB"><a href="sip:jon.morris@jbarisk.com"><span style="color:#F6A124;text-decoration:none">Skype</span></a><o:p></o:p></span></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p><strong><span style="color: #f6a125; font-family:arial;">T</span></strong><span style="text-decoration: none; color: #000000; font-family:arial;"> +44 (0) 1756 799919</span><br>
<a style="text-decoration: none; color: #f6a125; font-family:arial;" href="http://www.jbarisk.com">www.jbarisk.com</a></p>
<p><a href="http://www.jbarisk.com"><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-JBA.png" alt="Visit our website" width="33" height="26"></a> <a href="https://www.linkedin.com/in/jon-morris-a2897b4/"><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png" alt="" height="26"></a>
<a href="https://twitter.com/jbarisk"><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-TWITTER.png" alt="Follow us on Twitter" width="33" height="26"></a></p>
</body>
</html>