[GRASS-dev] r.watershed and swap memory

Tue Mar 18 20:50:00 EDT 2008

Hi Ivan and Helena,

149,040,000 Cells for the srtm 90m of Mekong river.
computer is on 8Gb RAM.
About 65 % done after 43.x hours, r.watershed basically does it, it seems.
Of course efficiency is a problem here, but it does it.

yes r.terraflow goes to /tmp, that is considered a bug to me, not an
enhancement...
The location from which we have the GRASS dataset is generally the
best guess for large empty Disk space.

Cheers,
Yann

On 19/03/2008, ivan marchesini <marchesini at unipg.it> wrote:
> Dear Helena,
>  Thank you very much for your answer...
>  my problems are:
>
>  * is the flowdirection output produced by r.terraflow suitable for basin
>  creation (like I can do with the drainage map created by r.watershed and
>  given as input to r.water.outlet)? Because this is my final target...
>  It seems to me that r.terraflow outputs aren't useful for basin
>  delineation... or I'm wrong?
>
>  * I'm really interested in testing TerraSTREAM but, probably due to my
>  fault, I wasn't able to obtain a login and password...
>
>  many many thanks
>
>  Ivan
>
>
>
>
>
>
>
>  Il giorno mar, 18/03/2008 alle 09.33 -0400, Helena Mitasova ha scritto:
>
> > Ivan - you may talk to Yann on this list before you buy more memory -
>  > he is trying to do the same as you , but with a bigger DEM and 8GB of
>  > memory (Yann I hope it is OK with you that I am revealing this here).
>  > My experience with large DEMs (up to 10,000x10,000) has been that I
>  > had to split the area into sections that were about
>  > 2000x2000 on 1GB memory computer) to get it done (I was able to do
>  > that for Panama because of its shape - many small watersheds rather
>  > than a single big one) - it took me several days to do that.
>  > Then I ran r.terraflow and I got it in 3 hours.
>  >
>  > Yann says that r.terraflow did not work for him - now I remember what
>  > the problem was when I tried to run it recently -
>  > it needs a LOT of hard drive space which is not a problem these days,
>  > BUT the default has been changed
>  > to /tmp which for my linux box is only 2GB or so. But when running
>  > r.terraflow you can define where
>  > you want the temporary files to be written - so give it something
>  > with a lot of space (tens of gigabytes at least)
>  > and it should run. I think that the default should be changed to
>  > where it was - I think it is the regular grass tmp where people
>  > usually have a lot of space for the data.
>  >
>  > If even that does not work you can give a try to brand new
>  > TerraSTREAM - see the link below
>  > (and let me know whether it works for you),
>  >
>  > Helena
>  >
>  >   TerraSTREAM provides a series of components that
>  > perform flow modeling and terrain analysis tasks on very large digital
>  > elevation models and works equally well on TIN and grid DEMs. The
>  > algorithms
>  > used in the libraries have provable efficient performance in the
>  > worst case,
>  > even on very large terrains that do not fit in the main memory of the
>  > computer.
>  > TerraSTREAM 0.2 comes with direct GRASS and ArcGIS support as well as
>  > a simple
>  > standalone graphical user interface and powerful command line tools
>  > that can be
>  > used alone or integrated into most GIS environments by scripting. For
>  > more
>  > information about this release and for contact information, visit
>  > http://madalgo.au.dk/Trac-TerraSTREAM/.
>  > The TerraSTREAM 0.2 users guide is available here:
>  > http://madalgo.au.dk/Trac-TerraSTREAM/wiki/UsersGuide .
>  >
>  >
>  >
>  > On Mar 18, 2008, at 5:32 AM, ivan marchesini wrote:
>  >
>  > > Dear Grass Users and Developers, sorry for cross posting but we
>  > > hope the
>  > > argument can be of interest for all and we hope someone can give us a
>  > > solution to this problem..
>  > >
>  > > We have this kind of problem:
>  > > * a large DEM (150000000 cells)
>  > > * an ordinary computer (2 GB ram)
>  > > * we must obtain the drainage map using r.watershed (without changing
>  > > resolution), because then we need to be able to calculate the upstream
>  > > basin for each cell (r.water.outlet).
>  > > * we have tried r.watershed straigth (but after few seconds a memory
>  > > allocation problem crashed the program)
>  > > * we have tried the "-m" option but after 4 days of work it is
>  > > still at
>  > > 0%.
>  > > * giving up the last option, because it takes too long, we have
>  > > monitored the ram usage by means of "free -m" and we have seen that
>  > > r.watershed rapidly saturate the ram and then, after a little usage of
>  > > swap (20 mb) crashes..  so it seems that r.watershed doesn't use swap
>  > > memory... (and is then unuseful, as we did, to increase the swap
>  > > memory)
>  > > * we have tried to modify
>  > > "Swappiness" (http://www.gentoo.it/doc/memory.html#doc_chap5) but
>  > > without success... the error is still the same
>  > >
>  > >
>  > > so at this point:
>  > > * is adding ram to the computer the only solution?
>  > > * if yes, how can we estimate the ram to buy!
>  > > * can we obtain some better results compiling grass with option
>  > > --enable-largefile
>  > > * does someone solved a similar problem in some other way?
>  > >
>  > > Thank you for your suggestions!!!
>  > > we hope really in your help
>  > >
>  > > Luca & Ivan
>  > >
>  > >
>  > >
>  > >
>  > >
>  > >
>  > >
>  > >
>  > >
>  > >
>  > >
>  > > --
>  > > Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
>  > > Preferisco formati liberi.
>  > > Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
>  > > I prefer free formats.
>  > > http://it.wikipedia.org/wiki/Formato_aperto
>  > > http://en.wikipedia.org/wiki/Open_format
>  > >
>  > > Ivan Marchesini
>  > > Department of Civil and Environmental Engineering
>  > > University of Perugia
>  > > Via G. Duranti 93/a
>  > > 06125
>  > > Perugia (Italy)
>  > > Socio fondatore GFOSS "Geospatial Free and Open Source Software"
>  > > http://www.gfoss.it
>  > > e-mail: marchesini at unipg.it
>  > >         ivan.marchesini at gmail.com
>  > > tel: +39(0)755853760
>  > > fax (university): +39(0)755853756
>  > > fax (home): +39(0)5782830887
>  > > jabber: geoivan73 at jabber.org
>  > >
>  > > _______________________________________________
>  > > grass-dev mailing list
>  > > grass-dev at lists.osgeo.org
>  > > http://lists.osgeo.org/mailman/listinfo/grass-dev
>  >
>  >
>  --
>  Ti prego di cercare di non inviarmi files .dwg, .doc, .xls, .ppt.
>  Preferisco formati liberi.
>  Please try to avoid to send me .dwg, .doc, .xls, .ppt files.
>  I prefer free formats.
>  http://it.wikipedia.org/wiki/Formato_aperto
>  http://en.wikipedia.org/wiki/Open_format
>
>  Ivan Marchesini
>  Department of Civil and Environmental Engineering
>  University of Perugia
>  Via G. Duranti 93/a
>  06125
>  Perugia (Italy)
>  Socio fondatore GFOSS "Geospatial Free and Open Source Software" http://www.gfoss.it
>  e-mail: marchesini at unipg.it
>         ivan.marchesini at gmail.com
>  tel: +39(0)755853760
>  fax (university): +39(0)755853756
>  fax (home): +39(0)5782830887
>  jabber: geoivan73 at jabber.org
>
>  _______________________________________________
>  grass-dev mailing list
>  grass-dev at lists.osgeo.org
>  http://lists.osgeo.org/mailman/listinfo/grass-dev
>