[gdal-dev] Raster size and ReadAsArray()

[Even Rouault]

Le mercredi 03 août 2011 15:35:22, Alexander Bruy a écrit :
> Hi,
> 
> There is a well-know "problem": reading really large rasters or bands
> into memory with DataSource.ReadAsArray() method impossible due
> memory limitations. For example, when I try to read one band with
> size 53109x29049 I get error:
> 
> File "C:\OSGeo4W\apps\Python25\lib\site-packages\osgeo\gdal.py", line
> 727, in ReadAsArray
>     return gdalnumeric.DatasetReadAsArray( self, xoff, yoff, xsize,
> ysize, buf_obj )
> File "C:\OSGeo4W\apps\Python25\lib\site-packages\osgeo\gdal_array.py",
> line 162, in DatasetReadAsArray
>     return BandReadAsArray( ds.GetRasterBand(1), xoff, yoff, xsize,
> ysize, buf_obj = buf_obj)
> File "C:\OSGeo4W\apps\Python25\lib\site-packages\osgeo\gdal_array.py",
> line 228, in BandReadAsArray
>     ar = numpy.empty([buf_ysize,buf_xsize], dtype = typecode)
> ValueError: dimensions too large.
> 
> I want to know is it possible to get maximum raster size that can be
> handled using ReadAsArray() without errors because I want to implement a
> fallback algorithm for large rasters in my tool.

It is difficult to predict in all cases. But there are a few things to keep in 
mind. Do you use a 32 bit build of GDAL or a 64 bit one ? If it is a 32 bit, 
then your memory allocations are generally limited to 2 GB by process, and 
much of the time even less because it is difficult to get reliably a continuous 
area of virtual memory of such a size. 53109 x 29049 = 1.5 GB if your data is 
of type Byte and if your dataset has only one band (multiply by the number of 
bands and the size in byte of the band data type). With a 64 bit build of GDAL 
and enough RAM, that should work more reliably.

> 
> Currently I try to implement it with try-except statement but maybe
> there is more
> elegant solution?

Yes, work with smaller buffers...

> 
> 
> Thanks