[GRASS-dev] r.watershed and swap memory

Tue Mar 18 09:33:23 EDT 2008

Ivan - you may talk to Yann on this list before you buy more memory -  
he is trying to do the same as you , but with a bigger DEM and 8GB of  
memory (Yann I hope it is OK with you that I am revealing this here).
My experience with large DEMs (up to 10,000x10,000) has been that I  
had to split the area into sections that were about
2000x2000 on 1GB memory computer) to get it done (I was able to do  
that for Panama because of its shape - many small watersheds rather  
than a single big one) - it took me several days to do that.
Then I ran r.terraflow and I got it in 3 hours.

Yann says that r.terraflow did not work for him - now I remember what  
the problem was when I tried to run it recently -
it needs a LOT of hard drive space which is not a problem these days,  
BUT the default has been changed
to /tmp which for my linux box is only 2GB or so. But when running  
r.terraflow you can define where
you want the temporary files to be written - so give it something  
with a lot of space (tens of gigabytes at least)
and it should run. I think that the default should be changed to  
where it was - I think it is the regular grass tmp where people  
usually have a lot of space for the data.

If even that does not work you can give a try to brand new  
TerraSTREAM - see the link below
(and let me know whether it works for you),

Helena

  TerraSTREAM provides a series of components that
perform flow modeling and terrain analysis tasks on very large digital
elevation models and works equally well on TIN and grid DEMs. The  
algorithms
used in the libraries have provable efficient performance in the  
worst case,
even on very large terrains that do not fit in the main memory of the  
computer.
TerraSTREAM 0.2 comes with direct GRASS and ArcGIS support as well as  
a simple
standalone graphical user interface and powerful command line tools  
that can be
used alone or integrated into most GIS environments by scripting. For  
more
information about this release and for contact information, visit
http://madalgo.au.dk/Trac-TerraSTREAM/.
The TerraSTREAM 0.2 users guide is available here:
http://madalgo.au.dk/Trac-TerraSTREAM/wiki/UsersGuide .

On Mar 18, 2008, at 5:32 AM, ivan marchesini wrote:

> Dear Grass Users and Developers, sorry for cross posting but we  
> hope the
> argument can be of interest for all and we hope someone can give us a
> solution to this problem..
>
> We have this kind of problem:
> * a large DEM (150000000 cells)
> * an ordinary computer (2 GB ram)
> * we must obtain the drainage map using r.watershed (without changing
> resolution), because then we need to be able to calculate the upstream
> basin for each cell (r.water.outlet).
> * we have tried r.watershed straigth (but after few seconds a memory
> allocation problem crashed the program)
> * we have tried the "-m" option but after 4 days of work it is  
> still at
> 0%.
> * giving up the last option, because it takes too long, we have
> monitored the ram usage by means of "free -m" and we have seen that
> r.watershed rapidly saturate the ram and then, after a little usage of
> swap (20 mb) crashes..  so it seems that r.watershed doesn't use swap
> memory... (and is then unuseful, as we did, to increase the swap
> memory)
> * we have tried to modify
> "Swappiness" (http://www.gentoo.it/doc/memory.html#doc_chap5) but
> without success... the error is still the same
>
>
> so at this point:
> * is adding ram to the computer the only solution?
> * if yes, how can we estimate the ram to buy!
> * can we obtain some better results compiling grass with option
> --enable-largefile
> * does someone solved a similar problem in some other way?
>
> Thank you for your suggestions!!!
> we hope really in your help
>
> Luca & Ivan
>
>
>
>
>
>
>
>
>
>
>
> -- 
> Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
> Preferisco formati liberi.
> Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
> I prefer free formats.
> http://it.wikipedia.org/wiki/Formato_aperto
> http://en.wikipedia.org/wiki/Open_format
>
> Ivan Marchesini
> Department of Civil and Environmental Engineering
> University of Perugia
> Via G. Duranti 93/a
> 06125
> Perugia (Italy)
> Socio fondatore GFOSS "Geospatial Free and Open Source Software"  
> http://www.gfoss.it
> e-mail: marchesini at unipg.it
>         ivan.marchesini at gmail.com
> tel: +39(0)755853760
> fax (university): +39(0)755853756
> fax (home): +39(0)5782830887
> jabber: geoivan73 at jabber.org
>
> _______________________________________________
> grass-dev mailing list
> grass-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-dev