<div dir="ltr">Hi Markus M, <div>I re-tested (size <span style="color:rgb(80,0,80);font-size:12.8px">80490 x 142690 = 11,485,118,100 </span>) r.stream.extract & r.stream.basins setting 25G RAM for the 2 commands with a slurm upper limit of 60G RAM.</div><div><br></div><div>The r.stream.extract use this RAM </div><div><div>############################################################</div><div> JobID MaxVMSize </div><div>------------ ---------- </div><div>6731334.bat+ 47459272K </div><div>############################################################</div></div><div>and finish without problem </div><div><br></div><div>rather r.stream.basins use </div><div><div>############################################################</div><div> JobID MaxVMSize </div><div>------------ ---------- </div><div>6731334.bat+ 90241708K </div><div>############################################################</div></div><div>and get kill with this error. </div><div><br></div><div><div>Reading raster map <stream>...</div><div>0..3..6..9..12..15../var/spool/slurmd/job6731334/slurm_script: line 88: 15041 Bus error /gpfs/home/fas/sbsc/ga254/.grass7/addons/bin/r.stream.basins -l stream_rast=stream direction=dir basins=lbasin memory=25000 --o --verbose</div></div><div><br></div><div>I think something is implemented different in r.stream.basins compare to r.stream.extract. </div><div>I will try to ask for more RAM (100G) but I'm afraid that is going to fail again. </div><div><br></div><div>Any thoughts?</div><div><br></div><div>Thanks you </div><div>Best </div><div>Giuseppe</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div> </div></div><div class="gmail_extra"><br><div class="gmail_quote">On 17 November 2017 at 03:33, Markus Metz <span dir="ltr"><<a href="mailto:markus.metz.giswork@gmail.com" target="_blank">markus.metz.giswork@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class=""><div><br><br>On Thu, Nov 16, 2017 at 10:51 PM, Giuseppe Amatulli <<a href="mailto:giuseppe.amatulli@gmail.com" target="_blank">giuseppe.amatulli@gmail.com</a>> wrote:<br>><br>><br>> Hi Markus M.<br>><br>> I was testing the r.stream.extract for 2 tiff <br>><br>> 1) 80040 x 72870 = 5,832,514,800 i got the stream results - no error <br></div><div><br></div></span><div>Great, that means large maps with more than 2 billion cells are supported.<br></div><span class=""><div>><br>> 2) 80490 x 142690 = 11,485,118,100 i got the following error <br>><br>> A* Search...<br>> 0..2../var/spool/slurmd/<wbr>job6514787/slurm_script: line 80: 28925 Bus error r.stream.extract elevation=elv accumulation=upa threshold=0.5 depression=dep direction=dir stream_raster=stream memory=45000 --o --verbose<br><br></div></span>From wikipedia:<br>"a bus error is a fault raised by hardware, notifying an operating system (OS) that a process is trying to access memory that the CPU cannot physically address: an invalid address for the address bus, hence the name."<br><div><br></div><div>I guess you either need to raise the RAM limit in slurm or slightly reduce the memory option for r.stream.extract.</div><div><br></div><div>Markus M<br></div><div><div class="h5"><div><br></div><div>><br>> Do you think that is something with the slurm ram limitation or is something with the r.extract.stream? <br>> If the the stream output have a number of stream segments larger than 2,147,483,647 what is happen? <br>> Do I get an error or all the value larger than 2,147,483,647 are just rounded to 2,147,483,647 ? <br>><br>><br>> Moreover, if use the stream obtain from option 1) and I use the stream as input for the r.stream.basins <br>> I got the following error<br>><br>> reading raster map <stream>...<br>> 0..3..6..9..12..15..18..21..<wbr>24..27..30..33..36..39..42..<wbr>45../var/spool/slurmd/<wbr>job6514788/slurm_script: line 82: 17687 Bus error /gpfs/home/fas/sbsc/ga254/.<wbr>grass7/addons/bin/r.stream.<wbr>basins -l stream_rast=stream direction=dir<br>><br>> is this something that need to be fixed in r.stream.basins, or should i think that is due to other problems <br>><br>> Thank you <br>> Giuseppe <br>><br>><br>><br>><br>> On 10 November 2017 at 11:21, Markus Neteler <<a href="mailto:neteler@osgeo.org" target="_blank">neteler@osgeo.org</a>> wrote:<br>>><br>>> On Wed, Nov 1, 2017 at 10:12 PM, Markus Metz<br>>> <<a href="mailto:markus.metz.giswork@gmail.com" target="_blank">markus.metz.giswork@gmail.com</a><wbr>> wrote:<br>>> > On Wed, Nov 1, 2017 at 7:15 PM, Giuseppe Amatulli<br>>> > <<a href="mailto:giuseppe.amatulli@gmail.com" target="_blank">giuseppe.amatulli@gmail.com</a>> wrote:<br>>> >><br>>> >> Thanks Markus!!<br>>> >> I will test and I will let you know how it works.<br>>> ><br>>> > Your feedback is very helpful!<br>>> >><br>>> >> I have few more questions<br>>> >> 1) now how much is the upper limit matrix cell number that<br>>> >> r.stream.extract can handle?<br>>> ><br>>> > About 1.15e+18 cells.<br>>> ><br>>> > Another limitation is the number of detected stream segments. This must not<br>>> > be larger than 2,147,483,647 streams,<br>>> ...<br>>><br>>> (Added as a note to<br>>> <a href="https://grasswiki.osgeo.org/wiki/GRASS_GIS_Performance#Some_benchmarks" target="_blank">https://grasswiki.osgeo.org/<wbr>wiki/GRASS_GIS_Performance#<wbr>Some_benchmarks</a><br>>> )<br>>><br>>> best,<br>>> markusN<br>><br>><br>><br>><br>> --<br>> Giuseppe Amatulli, Ph.D.<br>><br>> Research scientist at<br>> Yale School of Forestry & Environmental Studies<br>> Yale Center for Research Computing<br>> Center for Science and Social Science Information<br>> New Haven, 06511<br>> Teaching: <a href="http://spatial-ecology.net" target="_blank">http://spatial-ecology.net</a><br>> Work: <a href="https://environment.yale.edu/profile/giuseppe-amatulli/" target="_blank">https://environment.yale.edu/<wbr>profile/giuseppe-amatulli/</a><br><br></div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Giuseppe Amatulli, Ph.D.<br><br>Research scientist at<br><span>Yale School of Forestry & Environmental Studies<br></span><span>Yale Center for Research Computing<br></span>Center for Science and Social Science Information<br>New Haven, 06511<br><div>
Teaching: <a href="http://spatial-ecology.net" target="_blank">http://spatial-ecology.net</a></div>
Work: <a href="https://environment.yale.edu/profile/giuseppe-amatulli/" target="_blank">https://environment.yale.edu/profile/giuseppe-amatulli/</a> <br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>