[GRASS-user] r.stream.extract error

Sun Nov 26 23:31:07 PST 2017

On Mon, Nov 27, 2017 at 3:56 AM, Giuseppe Amatulli <
giuseppe.amatulli at gmail.com> wrote:
>
> Just finished the test with 100G and the computation finish without
error.
>
> So my conclusion is:
> even if setting the max value of RAM  in the  r.stream.basins  to
memory=25000 the effective requirements is a bit less the then 100G.

For r.stream.basins you need to use the -m flag, otherwise everything is
done in RAM. A nice enhancement for all affected r.stream.* modules would
be to switch automatically to disk swap mode if the amount of memory needed
for all-in-RAM processing is larger than what is allocated with the
memory=MB option.

> A similar situation happens with r.stream.extract where the requested
used RAM is a bit less then 50G.

This is strange. In my tests, r.stream.extract always uses a bit less than
what is given with the memory option. I could process a raster with
11,612,160,000 cells and memory=25000 and actual peak memory consumption
was about 24000 MB.

Markus M

>
> Any thoughts?
>
> Thanks you
> Best
> Giuseppe
>
>
>
>
>
>
> On 26 November 2017 at 20:33, Giuseppe Amatulli <
giuseppe.amatulli at gmail.com> wrote:
>>
>> Hi Markus M,
>> I re-tested (size 80490 x 142690 = 11,485,118,100 ) r.stream.extract &
r.stream.basins  setting 25G  RAM  for the 2 commands with a slurm upper
limit of 60G RAM.
>>
>> The r.stream.extract use this RAM
>> ############################################################
>>        JobID  MaxVMSize
>> ------------ ----------
>> 6731334.bat+  47459272K
>> ############################################################
>> and finish without problem
>>
>> rather r.stream.basins use
>> ############################################################
>>        JobID  MaxVMSize
>> ------------ ----------
>> 6731334.bat+  90241708K
>> ############################################################
>> and get kill with this error.
>>
>> Reading raster map <stream>...
>> 0..3..6..9..12..15../var/spool/slurmd/job6731334/slurm_script: line 88:
15041 Bus error
/gpfs/home/fas/sbsc/ga254/.grass7/addons/bin/r.stream.basins -l
stream_rast=stream direction=dir basins=lbasin memory=25000 --o --verbose
>>
>> I think something is implemented different in r.stream.basins compare to
r.stream.extract.
>> I will try to ask for more RAM (100G) but I'm afraid that is going to
fail again.
>>
>> Any thoughts?
>>
>> Thanks you
>> Best
>> Giuseppe
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On 17 November 2017 at 03:33, Markus Metz <markus.metz.giswork at gmail.com>
wrote:
>>>
>>>
>>>
>>> On Thu, Nov 16, 2017 at 10:51 PM, Giuseppe Amatulli <
giuseppe.amatulli at gmail.com> wrote:
>>> >
>>> >
>>> > Hi Markus  M.
>>> >
>>> > I was testing the r.stream.extract for 2 tiff
>>> >
>>> > 1) 80040 x 72870 = 5,832,514,800 i got the stream results - no error
>>>
>>> Great, that means large maps with more than 2 billion cells are
supported.
>>> >
>>> > 2) 80490 x 142690 = 11,485,118,100 i got the following error
>>> >
>>> > A* Search...
>>> > 0..2../var/spool/slurmd/job6514787/slurm_script: line 80: 28925 Bus
error               r.stream.extract elevation=elv accumulation=upa
threshold=0.5 depression=dep direction=dir stream_raster=stream
memory=45000 --o --verbose
>>>
>>> From wikipedia:
>>> "a bus error is a fault raised by hardware, notifying an operating
system (OS) that a process is trying to access memory that the CPU cannot
physically address: an invalid address for the address bus, hence the name."
>>>
>>> I guess you either need to raise the RAM limit in slurm or slightly
reduce the memory option for r.stream.extract.
>>>
>>> Markus M
>>>
>>> >
>>> > Do you think that is something with the slurm ram limitation or is
something with the r.extract.stream?
>>> > If the the stream output have a number of stream segments larger than
2,147,483,647 what is happen?
>>> > Do I get an error or all the value larger than   2,147,483,647 are
just rounded to 2,147,483,647 ?
>>> >
>>> >
>>> > Moreover, if use the stream obtain from option 1) and I use the
stream as input for the r.stream.basins
>>> > I got the following error
>>> >
>>> > reading raster map <stream>...
>>> >
0..3..6..9..12..15..18..21..24..27..30..33..36..39..42..45../var/spool/slurmd/job6514788/slurm_script:
line 82: 17687 Bus error
/gpfs/home/fas/sbsc/ga254/.grass7/addons/bin/r.stream.basins -l
stream_rast=stream direction=dir
>>> >
>>> > is this something that need to be fixed in r.stream.basins, or should
i think that is due to other problems
>>> >
>>> > Thank you
>>> > Giuseppe
>>> >
>>> >
>>> >
>>> >
>>> > On 10 November 2017 at 11:21, Markus Neteler <neteler at osgeo.org>
wrote:
>>> >>
>>> >> On Wed, Nov 1, 2017 at 10:12 PM, Markus Metz
>>> >> <markus.metz.giswork at gmail.com> wrote:
>>> >> > On Wed, Nov 1, 2017 at 7:15 PM, Giuseppe Amatulli
>>> >> > <giuseppe.amatulli at gmail.com> wrote:
>>> >> >>
>>> >> >> Thanks Markus!!
>>> >> >> I will test and I will let you know how it works.
>>> >> >
>>> >> > Your feedback is very helpful!
>>> >> >>
>>> >> >> I have few  more questions
>>> >> >> 1) now how much is the upper limit matrix cell number that
>>> >> >> r.stream.extract can handle?
>>> >> >
>>> >> > About 1.15e+18 cells.
>>> >> >
>>> >> > Another limitation is the number of detected stream segments. This
must not
>>> >> > be larger than 2,147,483,647 streams,
>>> >> ...
>>> >>
>>> >> (Added as a note to
>>> >>
https://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Some_benchmarks
>>> >> )
>>> >>
>>> >> best,
>>> >> markusN
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Giuseppe Amatulli, Ph.D.
>>> >
>>> > Research scientist at
>>> > Yale School of Forestry & Environmental Studies
>>> > Yale Center for Research Computing
>>> > Center for Science and Social Science Information
>>> > New Haven, 06511
>>> > Teaching: http://spatial-ecology.net
>>> > Work:  https://environment.yale.edu/profile/giuseppe-amatulli/
>>>
>>
>>
>>
>> --
>> Giuseppe Amatulli, Ph.D.
>>
>> Research scientist at
>> Yale School of Forestry & Environmental Studies
>> Yale Center for Research Computing
>> Center for Science and Social Science Information
>> New Haven, 06511
>> Teaching: http://spatial-ecology.net
>> Work:  https://environment.yale.edu/profile/giuseppe-amatulli/
>
>
>
>
> --
> Giuseppe Amatulli, Ph.D.
>
> Research scientist at
> Yale School of Forestry & Environmental Studies
> Yale Center for Research Computing
> Center for Science and Social Science Information
> New Haven, 06511
> Teaching: http://spatial-ecology.net
> Work:  https://environment.yale.edu/profile/giuseppe-amatulli/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20171127/f13c2b2d/attachment.html>