[GRASS-user] memory requirement of v.net.allpairs module (Markus
Metz)
Pankaj Kr Sharma
pkscwc at gmail.com
Fri Dec 30 10:56:19 EST 2011
Dear users and developers,
v.net.allpairs return +ve distance for points in the backward direction of
the vector and -ve value for points lying in the forward direction of the
vector.
(I have posted a similar script earlier for this purpose. Which was written
in bash script format.)
One can use these categories to sum up the areas (r.stream.basins) and
calculate the cumulative area without creating individual raster maps (
thus, using hard disk space) for each point.
This module is pretty fast in calculating the distance.
However, from the moment, when it writes the values in database (sequential
write , perhaps) , it's a slow process.
In the reply , it was mentioned that all the information (from cat, to cat,
distance ) is held in a temporary file. (reason for large swap space being
used 160 gb)
Can't we have a text file (from this temp file) instead of sequential
database write for large vector sets. (for example, as those created by
r.stats .)
The text file can be copied to database in few seconds. Whereas, this can
take six month (145,000 * 145,000).
I have run it on smaller region and on an vector output of 11000 points
(i.e. 11,000 * 11,000) ,it took approximately 12 hours to finish.
Can we have opinion of Dear Daniel Bundala ( the module developer ) on this.
Thanks.
On Wed, Dec 28, 2011 at 5:46 PM, <grass-user-request at lists.osgeo.org> wrote:
> Send grass-user mailing list submissions to
> grass-user at lists.osgeo.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.osgeo.org/mailman/listinfo/grass-user
> or, via email, send a message with subject or body 'help' to
> grass-user-request at lists.osgeo.org
>
> You can reach the person managing the list at
> grass-user-owner at lists.osgeo.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of grass-user digest..."
>
>
> Today's Topics:
>
> 1. Re: memory requirement of v.net.allpairs module (Markus Metz)
> 2. Compile and install 6.4.2RC2 (Aldo Clerici)
> 3. Re: Compile and install 6.4.2RC2 (Martin Landa)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 27 Dec 2011 18:53:20 +0100
> From: Markus Metz <markus.metz.giswork at googlemail.com>
> Subject: Re: [GRASS-user] memory requirement of v.net.allpairs module
> To: Pankaj Kr Sharma <pkscwc at gmail.com>
> Cc: grass-user at lists.osgeo.org, grass-dev at lists.osgeo.org
> Message-ID:
> <CAG+h=FGPOU61d45cQVw0UczvGWGJmVLmYYtqaaFPiD7mK-iKhw at mail.gmail.com
> >
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Sun, Dec 25, 2011 at 5:04 PM, Pankaj Kr Sharma <pkscwc at gmail.com>
> wrote:
> > Dear Developers and users,
> >
> > On a small test region (11000x11000 cells), with a threshold of 1000sqkm,
> > r.stream.extract resulted in a stream vector network of 190 points and
> > lines. The v.net.allpairs generated a table of distances between each
> point
> > within 10 seconds on my home machine (only 4gb ram). I thought that I
> got
> > my solution for finding sub-basins of a very large area. (Please refer
> to my
> > earlier posts in this regard.)
> >
> > On my work setup, (better machine, 16gb RAM , six cores), I followed this
> > routine:
> > 1. r.grow by one cell on area of interest. (It's an important step.
> Thanks
> > to Markus Metz for this tip.) Time taken by the module: approximately 2
> > hours.
> >
> > 2. r.stream.extract on my area of interest.(42000*42000 cells) with 100
> > sq.km. as threshold (12345 in number of cells.)
>
> Above you said that yourtest region was 11000x11000 cells, not
> 42000x42000 cells. Both r.grow and r.stream.extract will be
> considerably faster if you adjust your computational region to your
> test region.
>
> > Please note that the r.stream.extract module's vector output was a
> > network of 1,45,000 (approx) points and lines.
> > Time taken by module: approximately 24 hours.
> >
> > 3. Then I ran "v.net.allpairs" on this network. Now, v.net.allpairs is
> still
> > running. Approximately, one week has elapsed.
> > After three days, a message appeared on the console that says "writing
> data
> > into the table". After four days of this message, the progress bar is
> still
> > without any colour. ( Which I consider as progress being less than one
> > percent.)
> >
> > The other thing which I noticed was the memory consumption. The module
> has
> > consumed 15.5 gb ram out of 16gb, thereafter it consumed 166gb of swap
> space
> > out of 500gb swap space available.( Earlier, I had kept complete one
> disk of
> > 1 tb for swap.)
>
> This sounds like bad news. v.net.allpairs keeps all <from> categories,
> all <to> categories, and all distances from <from> to <to> in memory.
> For 145000 network nodes, this would be about 320 GB. Assuming that
> half of the features are lines and half of the features are points,
> v.net.allpairs would require about 160GB of memory, plus some overhead
> to handle the vector if you did not set the environment variable
> GRASS_VECTOR_LOWMEM. Granted that you did not run v.net.allpairs with
> the -a flag.
>
> Why did you use v.net.allpairs at all on the stream network?
>
> Markus M
>
> >
> > Some of the answers for this response of the module, I have myself.
> >
> > Like, the module has more calculations to do.
> > (145000*145000)/(190*190)~582410.
> > Still, as per my calculation of time required 582410*10
> seconds/(24*60*60)~
> > 6.75 days
> >
> > And, I should have my postgres table ready for further calculations.
> > But, still there is no sign of any progress.
> > Is the time and memory being consumed by the module as per design and ok.
> >
> > I can definitely sub-divide into basins and work further.
> >
> >
> > _______________________________________________
> > grass-user mailing list
> > grass-user at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/grass-user
> >
>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 28 Dec 2011 12:46:17 +0100
> From: "Aldo Clerici" <aldo.clerici at unipr.it>
> Subject: [GRASS-user] Compile and install 6.4.2RC2
> To: <grass-user at lists.osgeo.org>
> Message-ID: <045901ccc556$4f6c51e0$ee44f5a0$@clerici at unipr.it>
> Content-Type: text/plain; charset="us-ascii"
>
> Dear Developers and users,
>
> I'm having troubles in compiling and installing grass6.4.2RC2 on Fedora 10.
>
> I installed Python-2.7.2, wxPython-2.9.2.4, numy-1.6.1.
>
> I run: ./configure --with-cxx
> --with-python=/usr/local/bin/python2.7-config
> --with-wxwidgets=/usr/local/bin/wx-config
>
>
>
> with the following output:
>
> GRASS is now configured for: i686-pc-linux-gnu
>
>
>
> Source directory: /home/clerici/GRASSprograms/grass-6.4.2RC2
>
> Build directory: /home/clerici/GRASSprograms/grass-6.4.2RC2
>
> Installation directory: ${prefix}/grass-6.4.2RC2
>
> Startup script in directory: ${exec_prefix}/bin
>
> C compiler: gcc -g -O2
>
> C++ compiler: c++ -g -O2
>
> Building shared libraries: yes
>
> 64bit support: no
>
> OpenGL platform: X11
>
>
>
> MacOSX application: no
>
> MacOSX architectures:
>
> MacOSX SDK:
>
>
>
> Tcl/Tk NVIZ: yes
>
>
>
> BLAS support: no
>
> C++ support: yes
>
> Cairo support: no
>
> DWG support: no
>
> FFMPEG support: no
>
> FFTW support: yes
>
> FreeType support: no
>
> GDAL support: yes
>
> GEOS support: no
>
> GLw support: no
>
> JPEG support: yes
>
> LAPACK support: no
>
> Large File support (LFS): no
>
> Motif support: no
>
> MySQL support: no
>
> NLS support: no
>
> ODBC support: no
>
> OGR support: yes
>
> OpenGL support: yes
>
> PNG support: yes
>
> PostgreSQL support: no
>
> Python support: yes
>
> Readline support: no
>
> SQLite support: no
>
> Tcl/Tk support: yes
>
> wxWidgets support: yes
>
> TIFF support: yes
>
> X11 support: yes
>
>
>
> Then: make
>
> GRASS GIS compilation log
>
> -------------------------
>
> Started compilation: Wed Dec 28 12:22:59 CET 2011
>
> --
>
> Errors in:
>
> No errors detected.
>
> --
>
> Finished compilation: Wed Dec 28 12:24:11 CET 2011
>
>
>
>
>
> And su -c 'make install' that seems ok.
>
>
>
> However when I run: grass64 -wxpython I have the following error message:
>
>
>
> Cleaning up temporary files ...
>
> Starting GRASS ...
>
> ERROR: wxGUI requires wxPython. No module named wxversion
>
> Error in GUI startup. If necessary, please
>
> report this error to the GRASS developers.
>
> Switching to text mode now.
>
> Hit RETURN to continue...
>
>
>
> Where did I go wrong?
>
>
>
> Many thanks in advance.
>
>
>
> A. Clerici
>
> Parma University
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://lists.osgeo.org/pipermail/grass-user/attachments/20111228/d8363dca/attachment-0001.html
>
> ------------------------------
>
> Message: 3
> Date: Wed, 28 Dec 2011 13:16:29 +0100
> From: Martin Landa <landa.martin at gmail.com>
> Subject: Re: [GRASS-user] Compile and install 6.4.2RC2
> To: Aldo Clerici <aldo.clerici at unipr.it>
> Cc: grass-user at lists.osgeo.org
> Message-ID:
> <CA+Ei1Ocvg1hBsjQNt_bPA7ys2pzWiDWDBFRuuo3KD-WBXZg2Aw at mail.gmail.com
> >
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi,
>
> 2011/12/28 Aldo Clerici <aldo.clerici at unipr.it>:
> > ERROR: wxGUI requires wxPython. No module named wxversion
> >
> > Error in GUI startup. If necessary, please
> >
> > report this error to the GRASS developers.
> >
> > Switching to text mode now.
> >
> > Hit RETURN to continue...
> >
> >
> >
> > Where did I go wrong?
>
> you need to install package which contains `wxversion`. On Debian
> GNU/Linux it's eg. `python-wxversion`.
>
> Martin
>
> --
> Martin Landa <landa.martin gmail.com> * http://geo.fsv.cvut.cz/~landa
>
>
> ------------------------------
>
> _______________________________________________
> grass-user mailing list
> grass-user at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user
>
>
> End of grass-user Digest, Vol 68, Issue 48
> ******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/grass-user/attachments/20111230/3bfce648/attachment-0001.html
More information about the grass-user
mailing list