[GRASS-user] r.watershed and big files

[Micha Silver]

On Wed, 2011-08-24 at 13:58 +0200, arno at agerrius.nl wrote:
> Hello,
> 
> I am trying to use the r.watershed function to calculate streams (rivers)
> from elevation data. As long as I make sure that the GRASS region is not
> too big things work fine, but in the end I would like to do the
> calculations at the resolution of the elevation data.
> 
> Let me first clarify what I mean with big in this case. I can do the
> calculations with regions up to 10000x10000 pixels, but above that I start
> to run into trouble. My elevation data is about 80000x60000, so I guess
> that is quite big.
> 

There are some nice guidelines on the r.watershed man page regarding
memory requirements. Specifically:

------------------------------------------------------
In-memory mode and disk swap mode
There are two versions of this program: ram and seg. ram is used by
default, seg can be used by setting the -m flag. 
The ram version requires a maximum of 31 MB of RAM for 1 million cells.
Together with the amount of system memory (RAM) available, this value
can be used to estimate whether the current region can be processed with
the ram version. 
The ram version uses virtual memory managed by the operating system to
store all the data structures and is faster than the seg version; seg
uses the GRASS segmentation library which manages data in disk files.
seg uses only as much system memory (RAM) as specified with the memory
option, allowing other processes to operate on the same system, even
when the current geographic region is huge. 
Due to memory requirements of both programs, it is quite easy to run out
of memory when working with huge map regions. If the ram version runs
out of memory and the resolution size of the current geographic region
cannot be increased, either more memory needs to be added to the
computer, or the swap space size needs to be increased. If seg runs out
of memory, additional disk space needs to be freed up for the program to
run. The r.terraflow module was specifically designed with huge regions
in mind and may be useful here as an alternative.
---------------------------------------------------

So with your 80,000 X 60,000 = 4,800,000,000 cells region you would need
about 150 GB (!) of RAM to run it in memory.
So you'll most likely need to use the -m option which writes everything
out to disk, making the process much slower.
 


> So my first question would be: Is the file I try to process too big or has
> r.watershed been run successfully on such big files?
> 
> >From the mailing list I understand GRASS needs to be compiled with large
> file support to work with such big files. I am not sure if my version is, I
> have tried in on Windows with the stable version 6.4 from the OSGeo4W
> installer and on Ubuntu with the 6.4 version from their repository.
> 
> This is the error I got on Ubuntu while running r.watershed:
> 
> SECTION 3 beginning: Initiating Variables. 5 sections total.
> WARNING: No such file or directory
> WARNING: cseg_open(): could not write segment file
> Floating point exception
> WARNING: Subprocess failed with exit code 34817
> 

Not sure about these errors, THey do *not* indicate memory problems. 

> Any advice on how to prevent this error?
> 
> Thanks,
> 
> Arno
> _______________________________________________
> grass-user mailing list
> grass-user at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user
> 
> This mail was received via Mail-SeCure System.
> 
>