[GRASS-user] r.stream.extract error

Giuseppe Amatulli giuseppe.amatulli at gmail.com
Sun Nov 26 18:56:11 PST 2017


Just finished the test with 100G and the computation finish without error.

So my conclusion is:
even if setting the max value of RAM  in the  r.stream.basins
to memory=25000 the effective requirements is a bit less the then 100G.
A similar situation happens with r.stream.extract where the requested used
RAM is a bit less then 50G.

Any thoughts?

Thanks you
Best
Giuseppe






On 26 November 2017 at 20:33, Giuseppe Amatulli <giuseppe.amatulli at gmail.com
> wrote:

> Hi Markus M,
> I re-tested (size 80490 x 142690 = 11,485,118,100 ) r.stream.extract
> & r.stream.basins  setting 25G  RAM  for the 2 commands with a slurm upper
> limit of 60G RAM.
>
> The r.stream.extract use this RAM
> ############################################################
>        JobID  MaxVMSize
> ------------ ----------
> 6731334.bat+  47459272K
> ############################################################
> and finish without problem
>
> rather r.stream.basins use
> ############################################################
>        JobID  MaxVMSize
> ------------ ----------
> 6731334.bat+  90241708K
> ############################################################
> and get kill with this error.
>
> Reading raster map <stream>...
> 0..3..6..9..12..15../var/spool/slurmd/job6731334/slurm_script: line 88:
> 15041 Bus error               /gpfs/home/fas/sbsc/ga254/.
> grass7/addons/bin/r.stream.basins -l stream_rast=stream direction=dir
> basins=lbasin memory=25000 --o --verbose
>
> I think something is implemented different in r.stream.basins compare to
> r.stream.extract.
> I will try to ask for more RAM (100G) but I'm afraid that is going to fail
> again.
>
> Any thoughts?
>
> Thanks you
> Best
> Giuseppe
>
>
>
>
>
>
>
>
>
>
>
>
> On 17 November 2017 at 03:33, Markus Metz <markus.metz.giswork at gmail.com>
> wrote:
>
>>
>>
>> On Thu, Nov 16, 2017 at 10:51 PM, Giuseppe Amatulli <
>> giuseppe.amatulli at gmail.com> wrote:
>> >
>> >
>> > Hi Markus  M.
>> >
>> > I was testing the r.stream.extract for 2 tiff
>> >
>> > 1) 80040 x 72870 = 5,832,514,800 i got the stream results - no error
>>
>> Great, that means large maps with more than 2 billion cells are supported.
>> >
>> > 2) 80490 x 142690 = 11,485,118,100 i got the following error
>> >
>> > A* Search...
>> > 0..2../var/spool/slurmd/job6514787/slurm_script: line 80: 28925 Bus
>> error               r.stream.extract elevation=elv accumulation=upa
>> threshold=0.5 depression=dep direction=dir stream_raster=stream
>> memory=45000 --o --verbose
>>
>> From wikipedia:
>> "a bus error is a fault raised by hardware, notifying an operating system
>> (OS) that a process is trying to access memory that the CPU cannot
>> physically address: an invalid address for the address bus, hence the name."
>>
>> I guess you either need to raise the RAM limit in slurm or slightly
>> reduce the memory option for r.stream.extract.
>>
>> Markus M
>>
>> >
>> > Do you think that is something with the slurm ram limitation or is
>> something with the r.extract.stream?
>> > If the the stream output have a number of stream segments larger than
>> 2,147,483,647 what is happen?
>> > Do I get an error or all the value larger than   2,147,483,647 are just
>> rounded to 2,147,483,647 ?
>> >
>> >
>> > Moreover, if use the stream obtain from option 1) and I use the stream
>> as input for the r.stream.basins
>> > I got the following error
>> >
>> > reading raster map <stream>...
>> > 0..3..6..9..12..15..18..21..24..27..30..33..36..39..42..45..
>> /var/spool/slurmd/job6514788/slurm_script: line 82: 17687 Bus error
>>           /gpfs/home/fas/sbsc/ga254/.grass7/addons/bin/r.stream.basins
>> -l stream_rast=stream direction=dir
>> >
>> > is this something that need to be fixed in r.stream.basins, or should i
>> think that is due to other problems
>> >
>> > Thank you
>> > Giuseppe
>> >
>> >
>> >
>> >
>> > On 10 November 2017 at 11:21, Markus Neteler <neteler at osgeo.org> wrote:
>> >>
>> >> On Wed, Nov 1, 2017 at 10:12 PM, Markus Metz
>> >> <markus.metz.giswork at gmail.com> wrote:
>> >> > On Wed, Nov 1, 2017 at 7:15 PM, Giuseppe Amatulli
>> >> > <giuseppe.amatulli at gmail.com> wrote:
>> >> >>
>> >> >> Thanks Markus!!
>> >> >> I will test and I will let you know how it works.
>> >> >
>> >> > Your feedback is very helpful!
>> >> >>
>> >> >> I have few  more questions
>> >> >> 1) now how much is the upper limit matrix cell number that
>> >> >> r.stream.extract can handle?
>> >> >
>> >> > About 1.15e+18 cells.
>> >> >
>> >> > Another limitation is the number of detected stream segments. This
>> must not
>> >> > be larger than 2,147,483,647 streams,
>> >> ...
>> >>
>> >> (Added as a note to
>> >>  https://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Some
>> _benchmarks
>> >> )
>> >>
>> >> best,
>> >> markusN
>> >
>> >
>> >
>> >
>> > --
>> > Giuseppe Amatulli, Ph.D.
>> >
>> > Research scientist at
>> > Yale School of Forestry & Environmental Studies
>> > Yale Center for Research Computing
>> > Center for Science and Social Science Information
>> > New Haven, 06511
>> > Teaching: http://spatial-ecology.net
>> > Work:  https://environment.yale.edu/profile/giuseppe-amatulli/
>>
>>
>
>
> --
> Giuseppe Amatulli, Ph.D.
>
> Research scientist at
> Yale School of Forestry & Environmental Studies
> Yale Center for Research Computing
> Center for Science and Social Science Information
> New Haven, 06511
> Teaching: http://spatial-ecology.net
> Work:  https://environment.yale.edu/profile/giuseppe-amatulli/
>



-- 
Giuseppe Amatulli, Ph.D.

Research scientist at
Yale School of Forestry & Environmental Studies
Yale Center for Research Computing
Center for Science and Social Science Information
New Haven, 06511
Teaching: http://spatial-ecology.net
Work:  https://environment.yale.edu/profile/giuseppe-amatulli/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20171126/3a124bee/attachment-0001.html>


More information about the grass-user mailing list