<span class="Apple-style-span" style="border-collapse: collapse; font-family: arial, sans-serif; font-size: 13px; ">I have been working on this problem as well. Initially, the attempt was to ReadAsArray small chunks. Unfortunately this is quite inefficient. Someone more knowledgeable will know why, but I suspect it has to do with either thrashing or the fact that full blocks are not being read in (as is the case when a 5000x5000 pixel block is read in on a 12567, 12764 GTiff).<div>
<br></div><div>My intention this morning is to try to implement an if statement which checks total file size and breaks it down into manageable (500MB maybe) chunks. Then using numpy slices, I can read, slice, mask, and manipulate as needed.</div>
<div><br></div><div>Again, the biggest issue I have had with this issue is performance. Profiling shows that _gdal.IORaster is utilizing a ton of CPU time.</div><div><br></div><div>Hope that helps, </div><div>Jay</div></span><br>
<div class="gmail_quote">On Wed, Aug 3, 2011 at 7:39 AM, Antonio Valentino <span dir="ltr"><<a href="mailto:antonio.valentino@tiscali.it">antonio.valentino@tiscali.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi Alexander,<br>
<br>
Il 03/08/2011 15:35, Alexander Bruy ha scritto:<br>
<div class="im">> Hi,<br>
><br>
> There is a well-know "problem": reading really large rasters or bands<br>
> into memory with DataSource.ReadAsArray() method impossible due<br>
> memory limitations. For example, when I try to read one band with<br>
> size 53109x29049 I get error:<br>
<br>
</div>[CUT]<br>
<div class="im"><br>
> I want to know is it possible to get maximum raster size that can be handled<br>
> using ReadAsArray() without errors because I want to implement a fallback<br>
> algorithm for large rasters in my tool.<br>
<br>
</div>In my experience using too large chunks of memory can cause,<br>
paradoxically, slowdowns.<br>
My suggestion is to define a reasonable maximum size for arrays in your<br>
application/library and switch to the "fallback algorithm for large<br>
rasters" every time that MAX_SIZE is exceeded, even if using ReadAsArray<br>
still works.<br>
<div class="im"><br>
> Currently I try to implement it with try-except statement but maybe<br>
> there is more<br>
> elegant solution?<br>
><br>
><br>
> Thanks<br>
<br>
</div>IMHO using try-except *is* elegant and perfectly in line with python's<br>
philosophy.<br>
<br>
best regards<br>
<br>
--<br>
<font color="#888888">Antonio Valentino<br>
</font><div><div></div><div class="h5">_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">http://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</div></div></blockquote></div><br>