[GRASS-user] r.stream.extract error

[Giuseppe Amatulli]

Hi Markus M,
I re-tested (size 80490 x 142690 = 11,485,118,100 ) r.stream.extract
& r.stream.basins  setting 25G  RAM  for the 2 commands with a slurm upper
limit of 60G RAM.

The r.stream.extract use this RAM
############################################################
       JobID  MaxVMSize
------------ ----------
6731334.bat+  47459272K
############################################################
and finish without problem

rather r.stream.basins use
############################################################
       JobID  MaxVMSize
------------ ----------
6731334.bat+  90241708K
############################################################
and get kill with this error.

Reading raster map <stream>...
0..3..6..9..12..15../var/spool/slurmd/job6731334/slurm_script: line 88:
15041 Bus error
 /gpfs/home/fas/sbsc/ga254/.grass7/addons/bin/r.stream.basins -l
stream_rast=stream direction=dir basins=lbasin memory=25000 --o --verbose

I think something is implemented different in r.stream.basins compare to
r.stream.extract.
I will try to ask for more RAM (100G) but I'm afraid that is going to fail
again.

Any thoughts?

Thanks you
Best
Giuseppe












On 17 November 2017 at 03:33, Markus Metz <markus.metz.giswork at gmail.com>
wrote:

>
>
> On Thu, Nov 16, 2017 at 10:51 PM, Giuseppe Amatulli <
> giuseppe.amatulli at gmail.com> wrote:
> >
> >
> > Hi Markus  M.
> >
> > I was testing the r.stream.extract for 2 tiff
> >
> > 1) 80040 x 72870 = 5,832,514,800 i got the stream results - no error
>
> Great, that means large maps with more than 2 billion cells are supported.
> >
> > 2) 80490 x 142690 = 11,485,118,100 i got the following error
> >
> > A* Search...
> > 0..2../var/spool/slurmd/job6514787/slurm_script: line 80: 28925 Bus
> error               r.stream.extract elevation=elv accumulation=upa
> threshold=0.5 depression=dep direction=dir stream_raster=stream
> memory=45000 --o --verbose
>
> From wikipedia:
> "a bus error is a fault raised by hardware, notifying an operating system
> (OS) that a process is trying to access memory that the CPU cannot
> physically address: an invalid address for the address bus, hence the name."
>
> I guess you either need to raise the RAM limit in slurm or slightly reduce
> the memory option for r.stream.extract.
>
> Markus M
>
> >
> > Do you think that is something with the slurm ram limitation or is
> something with the r.extract.stream?
> > If the the stream output have a number of stream segments larger than
> 2,147,483,647 what is happen?
> > Do I get an error or all the value larger than   2,147,483,647 are just
> rounded to 2,147,483,647 ?
> >
> >
> > Moreover, if use the stream obtain from option 1) and I use the stream
> as input for the r.stream.basins
> > I got the following error
> >
> > reading raster map <stream>...
> > 0..3..6..9..12..15..18..21..24..27..30..33..36..39..42..
> 45../var/spool/slurmd/job6514788/slurm_script: line 82: 17687 Bus error
>             /gpfs/home/fas/sbsc/ga254/.grass7/addons/bin/r.stream.basins
> -l stream_rast=stream direction=dir
> >
> > is this something that need to be fixed in r.stream.basins, or should i
> think that is due to other problems
> >
> > Thank you
> > Giuseppe
> >
> >
> >
> >
> > On 10 November 2017 at 11:21, Markus Neteler <neteler at osgeo.org> wrote:
> >>
> >> On Wed, Nov 1, 2017 at 10:12 PM, Markus Metz
> >> <markus.metz.giswork at gmail.com> wrote:
> >> > On Wed, Nov 1, 2017 at 7:15 PM, Giuseppe Amatulli
> >> > <giuseppe.amatulli at gmail.com> wrote:
> >> >>
> >> >> Thanks Markus!!
> >> >> I will test and I will let you know how it works.
> >> >
> >> > Your feedback is very helpful!
> >> >>
> >> >> I have few  more questions
> >> >> 1) now how much is the upper limit matrix cell number that
> >> >> r.stream.extract can handle?
> >> >
> >> > About 1.15e+18 cells.
> >> >
> >> > Another limitation is the number of detected stream segments. This
> must not
> >> > be larger than 2,147,483,647 streams,
> >> ...
> >>
> >> (Added as a note to
> >>  https://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Some_benchmarks
> >> )
> >>
> >> best,
> >> markusN
> >
> >
> >
> >
> > --
> > Giuseppe Amatulli, Ph.D.
> >
> > Research scientist at
> > Yale School of Forestry & Environmental Studies
> > Yale Center for Research Computing
> > Center for Science and Social Science Information
> > New Haven, 06511
> > Teaching: http://spatial-ecology.net
> > Work:  https://environment.yale.edu/profile/giuseppe-amatulli/
>
>


-- 
Giuseppe Amatulli, Ph.D.

Research scientist at
Yale School of Forestry & Environmental Studies
Yale Center for Research Computing
Center for Science and Social Science Information
New Haven, 06511
Teaching: http://spatial-ecology.net
Work:  https://environment.yale.edu/profile/giuseppe-amatulli/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20171126/eec468c5/attachment.html>