[gdal-dev] Raster algebra with Gdal and Python

[Dylan Beaudette]

On Tuesday 02 February 2010, Vincent Schut wrote:
> On 02/02/2010 08:01 AM, Alexander Bruy wrote:
> > Hi again,
> >
> > with your help I've fix problem with "integer overflow" and now I get
> > correct results.
> >
> > Also I clean up my code for reduce memory usage but problem with large
> > files is remain. I have raster with size 11779*10663 and 5 bands. When I
> > calculate NDVI on Linux box (2 Gb RAM, 2Gb swap, GDAL 1.6.3) all works
> > fine and I get correct results.Peak memory consumption during this
> > operation is 1900 Mb RAM and 300 Mb swap (monitored with htop).
> >
> > When I run same code with same raster on Windows box (2 Gb RAM, swap size
> > is selected by OS ~1.3 Gb, GDAL 1.6.3) I get MemoryError. Peak memory
> > consumption during this operation is 600 Mb RAM and 700 Mb swap. I've try
> > smaller raster with size
> > 8500*7600 pixels but get MemoryError too.
>
> Alexander,
>
> if you want your raster algebra to be independent of raster size, you'll
> have to resort to do the processing in a loop, each time processing a
> part of the input(s), writing the result, and so on.
> Easiest would be (to program) to loop over the image lines. Read a line
> for each input, calculate the resulting line, write that line to the
> output, and repeat.
> Performance-wise, it could be better to calculate in blocks of lines,
> for example 50 lines per iteration.
> This of course gets more complicated when your calculations include
> neighbourhood expressions.
>
> Vincent.
>

Not to rain on anyone's parade... but this type of operation tends to scale 
much better in tools that are row-based, and not memory bound, i.e. GRASS. 

Cheers,
Dylan


-- 
Dylan Beaudette
Soil Resource Laboratory
http://casoilresource.lawr.ucdavis.edu/
University of California at Davis
530.754.7341