[GRASS-user] r.watershed and big files

[Markus Metz]

On Wed, Aug 24, 2011 at 3:49 PM, Micha Silver <micha at arava.co.il> wrote:
> On Wed, 2011-08-24 at 13:58 +0200, arno at agerrius.nl wrote:
>> Hello,
>>
>> I am trying to use the r.watershed function to calculate streams (rivers)
>> from elevation data. As long as I make sure that the GRASS region is not
>> too big things work fine, but in the end I would like to do the
>> calculations at the resolution of the elevation data.
>>
>> Let me first clarify what I mean with big in this case. I can do the
>> calculations with regions up to 10000x10000 pixels, but above that I start
>> to run into trouble. My elevation data is about 80000x60000, so I guess
>> that is quite big.
>>
>
> There are some nice guidelines on the r.watershed man page regarding
> memory requirements. Specifically:
>
> ------------------------------------------------------
> In-memory mode and disk swap mode
> There are two versions of this program: ram and seg. ram is used by
> default, seg can be used by setting the -m flag.
> The ram version requires a maximum of 31 MB of RAM for 1 million cells.
> Together with the amount of system memory (RAM) available, this value
> can be used to estimate whether the current region can be processed with
> the ram version.
> The ram version uses virtual memory managed by the operating system to
> store all the data structures and is faster than the seg version; seg
> uses the GRASS segmentation library which manages data in disk files.
> seg uses only as much system memory (RAM) as specified with the memory
> option, allowing other processes to operate on the same system, even
> when the current geographic region is huge.
> Due to memory requirements of both programs, it is quite easy to run out
> of memory when working with huge map regions. If the ram version runs
> out of memory and the resolution size of the current geographic region
> cannot be increased, either more memory needs to be added to the
> computer, or the swap space size needs to be increased. If seg runs out
> of memory, additional disk space needs to be freed up for the program to
> run. The r.terraflow module was specifically designed with huge regions
> in mind and may be useful here as an alternative.
> ---------------------------------------------------
>
> So with your 80,000 X 60,000 = 4,800,000,000 cells region you would need
> about 150 GB (!) of RAM to run it in memory.

This large number of cells is not yet supported in r.watershed (see
similar problem with r.terraflow [0]). The maximum number of cells is
2^31 - 1 = 2,147,483,647. This limitation is 1) independent of whether
the in-memory or on-disk option is used, 2) for the (recommended)
on-disk option mainly imposed by the segment library of grass and
applies therefore also to e.g r.cost and r.walk.

> So you'll most likely need to use the -m option which writes everything
> out to disk, making the process much slower.
>
>
>
>> So my first question would be: Is the file I try to process too big or has
>> r.watershed been run successfully on such big files?
>>
>> >From the mailing list I understand GRASS needs to be compiled with large
>> file support to work with such big files. I am not sure if my version is, I
>> have tried in on Windows with the stable version 6.4 from the OSGeo4W
>> installer and on Ubuntu with the 6.4 version from their repository.
>>
>> This is the error I got on Ubuntu while running r.watershed:
>>
>> SECTION 3 beginning: Initiating Variables. 5 sections total.
>> WARNING: No such file or directory
>> WARNING: cseg_open(): could not write segment file
>> Floating point exception
>> WARNING: Subprocess failed with exit code 34817
>>
>
> Not sure about these errors, THey do *not* indicate memory problems.
>
I guess so too. The first message indicates that the on-disk option
(-m flag) is used. One reason could be not enough free disk space to
create temporary files. BTW, there seems to be a cut'n paste error,
the first message should read "SECTION 1 beginning: ..." and not
"SECTION 3 beginning: ..."

Just out of curiosity: how did you create this massive 4.8 billion cells DEM?

Markus M

[0] http://lists.osgeo.org/pipermail/grass-dev/2011-August/055314.html


>> Any advice on how to prevent this error?
>>
>> Thanks,
>>
>> Arno
>> _______________________________________________
>> grass-user mailing list
>> grass-user at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/grass-user
>>
>> This mail was received via Mail-SeCure System.
>>
>>
>
>
> _______________________________________________
> grass-user mailing list
> grass-user at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user
>