[GRASS-user] GRASS 7 error code 9

[Markus Metz]

Pankaj Kr Sharma wrote:
> Dear  Grass users and developers,
>
> The region is only 42001 * 42001 i.e. smaller than the limit of 46341 x
> 46341.
> The module r.watershed didn't work without -m due to memory requirement.
> So, the next best, I thought was to assign RAM as much as possible.
> And I began trying with 15500 MB downwards.

It is not a good idea to assign all free memory to r.watershed -m,
this will actually make the module slower. If there are e.g. a total
of 16 GB RAM, some of that is used by the os and possibly other
applications, so there may be only 15 GB free (currently unused). The
memory option should be set to some value smaller than the currently
available RAM, otherwise the module will become much slower when going
into swap space. The idea of the -m flag and memory option is to
prevent r.watershed from using up all memory. As a rough rule of
thumb, the memory option should be set to 50% - 80% of what free -m
reports as free. Anything larger would probably slow the module down.

Markus M


> I understand that it's really difficult to isolate Memory overflow errors in
> any program code.
> Both the modules have worked ( r.watershed and r.stream.extract) after
> allocating approximately half the available RAM.
> I am really interested in investigating the reasons of r.stream.extract not
> creating the vector table in postgres with large regions.
> Please note that for smaller regions, everything is perfect.
> I have observed that the initial calculation of memory requirement is not
> perfect but it's a good indicator.
> Almost two years back with quadcore AMD, 2 gb RAM on GRASSv6 , I had
> observed this.
> For a much smaller region and 1 kM resolution data, the memory required as
> reported by r.watershed was 60gb (without RAM i.e. -m flag).
> With usual swap space of 4gb, it didn't worked.
> However, it worked when I added a temporary swap space of 38 gb and used -m
> flag.
> I ran my computer uninterrupted for a week to get the results.
>
> The GRASSv7 is in much better shape and moving in right direction to utilise
> fully the expected hardware development (processing power, cheaper RAM,
> multiple cores)
>  and better quality of DEM data. ( Eight years back, it was a big pain
> digitising the contours for the fun of watching rivers flow on your
> desktop.)
>
> Now, in the coming month, what I plan that I will format the computer.
> Reload everything from scratch.
> And on 1 tb swap space and 1 tb disk space , try everything afresh.
> This will highlight the issue related with disk space, if any.
> I will additionally try to do things advised by GRASS  developers.
>
>
> On Sun, Nov 20, 2011 at 3:38 AM, Glynn Clements <glynn at gclements.plus.com>
> wrote:
>>
>> Hamish wrote:
>>
>> > note that at a region size of 46341 x 46341 we
>> > get to 2^31 cells, and to the point where a signed
>> > 32bit integer overflows and wraps backwards on
>> > itself. If that were the case I'd suspect a malloc
>> > error or a segfault not SIGKILL, but it may be a
>> > clue.
>>
>> The value of 9 indicates that the child process (etc/r.watershed.seg)
>> terminated either with exit(9) or due to signal 9 (SIGKILL). The
>> latter seems more likely (i.e. SIGKILL from the kernel's "OOM-killer"
>> is quite likely for a process which uses too much memory, while I
>> can't see any mechanism by which exit(9) would occur).
>>
>> If the kernel runs low on either physical or virtual memory, it
>> identifies a process which is using a lot of memory and kills it as if
>> by sending SIGKILL (although SIGKILL isn't actually "sent"; it can't
>> be blocked, ignored or caught, so the kernel just deletes the process;
>> if the parent calls wait() etc on the process, it is reported that the
>> process was terminated via SIGKILL).
>>
>> AFAICT, r.watershed requires far more memory than just the size of the
>> underlying raster data. It's possible that it isn't interpreting the
>> memory= parameter correctly. It's also possible that the kernel is
>> including the memory used for caching the segment file in deciding
>> which process to kill.
>>
>> --
>> Glynn Clements <glynn at gclements.plus.com>
>
>