Dear users and developers,<br><br>v.net.allpairs return +ve distance for points in the backward direction of the vector and -ve value for points lying in the forward direction of the vector.<br>(I have posted a similar script earlier for this purpose. Which was written in bash script format.)<br>
One can use these categories to sum up the areas (r.stream.basins) and calculate the cumulative area without creating individual raster maps ( thus, using hard disk space) for each point.<br>This module is pretty fast in calculating the distance.<br>
However, from the moment, when it writes the values in database (sequential write , perhaps) , it's a slow process.<br>In the reply , it was mentioned that all the information (from cat, to cat, distance ) is held in a temporary file. (reason for large swap space being used 160 gb)<br>
Can't we have a text file (from this temp file) instead of sequential database write for large vector sets. (for example, as those created by r.stats .)<br>The text file can be copied to database in few seconds. Whereas, this can take six month (145,000 * 145,000).<br>
I have run it on smaller region and on an vector output of 11000 points (i.e. 11,000 * 11,000) ,it took approximately 12 hours to finish. <br>Can we have opinion of Dear Daniel Bundala ( the module developer ) on this.<br>
<br>Thanks.<br><br><div class="gmail_quote">On Wed, Dec 28, 2011 at 5:46 PM, <span dir="ltr"><<a href="mailto:grass-user-request@lists.osgeo.org">grass-user-request@lists.osgeo.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Send grass-user mailing list submissions to<br>
<a href="mailto:grass-user@lists.osgeo.org">grass-user@lists.osgeo.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.osgeo.org/mailman/listinfo/grass-user" target="_blank">http://lists.osgeo.org/mailman/listinfo/grass-user</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:grass-user-request@lists.osgeo.org">grass-user-request@lists.osgeo.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:grass-user-owner@lists.osgeo.org">grass-user-owner@lists.osgeo.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of grass-user digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: memory requirement of v.net.allpairs module (Markus Metz)<br>
2. Compile and install 6.4.2RC2 (Aldo Clerici)<br>
3. Re: Compile and install 6.4.2RC2 (Martin Landa)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Tue, 27 Dec 2011 18:53:20 +0100<br>
From: Markus Metz <<a href="mailto:markus.metz.giswork@googlemail.com">markus.metz.giswork@googlemail.com</a>><br>
Subject: Re: [GRASS-user] memory requirement of v.net.allpairs module<br>
To: Pankaj Kr Sharma <<a href="mailto:pkscwc@gmail.com">pkscwc@gmail.com</a>><br>
Cc: <a href="mailto:grass-user@lists.osgeo.org">grass-user@lists.osgeo.org</a>, <a href="mailto:grass-dev@lists.osgeo.org">grass-dev@lists.osgeo.org</a><br>
Message-ID:<br>
<CAG+h=<a href="mailto:FGPOU61d45cQVw0UczvGWGJmVLmYYtqaaFPiD7mK-iKhw@mail.gmail.com">FGPOU61d45cQVw0UczvGWGJmVLmYYtqaaFPiD7mK-iKhw@mail.gmail.com</a>><br>
Content-Type: text/plain; charset=ISO-8859-1<br>
<br>
On Sun, Dec 25, 2011 at 5:04 PM, Pankaj Kr Sharma <<a href="mailto:pkscwc@gmail.com">pkscwc@gmail.com</a>> wrote:<br>
> Dear Developers and users,<br>
><br>
> On a small test region (11000x11000 cells), with a threshold of 1000sqkm,<br>
> r.stream.extract resulted in a stream vector network of 190 points and<br>
> lines. The v.net.allpairs generated a table of distances between each point<br>
> within 10 seconds on my home machine (only 4gb ram). I thought that I got<br>
> my solution for finding sub-basins of a very large area. (Please refer to my<br>
> earlier posts in this regard.)<br>
><br>
> On my work setup, (better machine, 16gb RAM , six cores), I followed this<br>
> routine:<br>
> 1. r.grow by one cell on area of interest. (It's an important step. Thanks<br>
> to Markus Metz for this tip.) Time taken by the module: approximately 2<br>
> hours.<br>
><br>
> 2. r.stream.extract on my area of interest.(42000*42000 cells) with 100<br>
> <a href="http://sq.km" target="_blank">sq.km</a>. as threshold (12345 in number of cells.)<br>
<br>
Above you said that yourtest region was 11000x11000 cells, not<br>
42000x42000 cells. Both r.grow and r.stream.extract will be<br>
considerably faster if you adjust your computational region to your<br>
test region.<br>
<br>
> Please note that the r.stream.extract module's vector output was a<br>
> network of 1,45,000 (approx) points and lines.<br>
> Time taken by module: approximately 24 hours.<br>
><br>
> 3. Then I ran "v.net.allpairs" on this network. Now, v.net.allpairs is still<br>
> running. Approximately, one week has elapsed.<br>
> After three days, a message appeared on the console that says "writing data<br>
> into the table". After four days of this message, the progress bar is still<br>
> without any colour. ( Which I consider as progress being less than one<br>
> percent.)<br>
><br>
> The other thing which I noticed was the memory consumption. The module has<br>
> consumed 15.5 gb ram out of 16gb, thereafter it consumed 166gb of swap space<br>
> out of 500gb swap space available.( Earlier, I had kept complete one disk of<br>
> 1 tb for swap.)<br>
<br>
This sounds like bad news. v.net.allpairs keeps all <from> categories,<br>
all <to> categories, and all distances from <from> to <to> in memory.<br>
For 145000 network nodes, this would be about 320 GB. Assuming that<br>
half of the features are lines and half of the features are points,<br>
v.net.allpairs would require about 160GB of memory, plus some overhead<br>
to handle the vector if you did not set the environment variable<br>
GRASS_VECTOR_LOWMEM. Granted that you did not run v.net.allpairs with<br>
the -a flag.<br>
<br>
Why did you use v.net.allpairs at all on the stream network?<br>
<br>
Markus M<br>
<br>
><br>
> Some of the answers for this response of the module, I have myself.<br>
><br>
> Like, the module has more calculations to do.<br>
> (145000*145000)/(190*190)~582410.<br>
> Still, as per my calculation of time required 582410*10 seconds/(24*60*60)~<br>
> 6.75 days<br>
><br>
> And, I should have my postgres table ready for further calculations.<br>
> But, still there is no sign of any progress.<br>
> Is the time and memory being consumed by the module as per design and ok.<br>
><br>
> I can definitely sub-divide into basins and work further.<br>
><br>
><br>
> _______________________________________________<br>
> grass-user mailing list<br>
> <a href="mailto:grass-user@lists.osgeo.org">grass-user@lists.osgeo.org</a><br>
> <a href="http://lists.osgeo.org/mailman/listinfo/grass-user" target="_blank">http://lists.osgeo.org/mailman/listinfo/grass-user</a><br>
><br>
<br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Wed, 28 Dec 2011 12:46:17 +0100<br>
From: "Aldo Clerici" <<a href="mailto:aldo.clerici@unipr.it">aldo.clerici@unipr.it</a>><br>
Subject: [GRASS-user] Compile and install 6.4.2RC2<br>
To: <<a href="mailto:grass-user@lists.osgeo.org">grass-user@lists.osgeo.org</a>><br>
Message-ID: <045901ccc556$4f6c51e0$ee44f5a0$@<a href="mailto:clerici@unipr.it">clerici@unipr.it</a>><br>
Content-Type: text/plain; charset="us-ascii"<br>
<br>
Dear Developers and users,<br>
<br>
I'm having troubles in compiling and installing grass6.4.2RC2 on Fedora 10.<br>
<br>
I installed Python-2.7.2, wxPython-2.9.2.4, numy-1.6.1.<br>
<br>
I run: ./configure --with-cxx --with-python=/usr/local/bin/python2.7-config<br>
--with-wxwidgets=/usr/local/bin/wx-config<br>
<br>
<br>
<br>
with the following output:<br>
<br>
GRASS is now configured for: i686-pc-linux-gnu<br>
<br>
<br>
<br>
Source directory: /home/clerici/GRASSprograms/grass-6.4.2RC2<br>
<br>
Build directory: /home/clerici/GRASSprograms/grass-6.4.2RC2<br>
<br>
Installation directory: ${prefix}/grass-6.4.2RC2<br>
<br>
Startup script in directory: ${exec_prefix}/bin<br>
<br>
C compiler: gcc -g -O2<br>
<br>
C++ compiler: c++ -g -O2<br>
<br>
Building shared libraries: yes<br>
<br>
64bit support: no<br>
<br>
OpenGL platform: X11<br>
<br>
<br>
<br>
MacOSX application: no<br>
<br>
MacOSX architectures:<br>
<br>
MacOSX SDK:<br>
<br>
<br>
<br>
Tcl/Tk NVIZ: yes<br>
<br>
<br>
<br>
BLAS support: no<br>
<br>
C++ support: yes<br>
<br>
Cairo support: no<br>
<br>
DWG support: no<br>
<br>
FFMPEG support: no<br>
<br>
FFTW support: yes<br>
<br>
FreeType support: no<br>
<br>
GDAL support: yes<br>
<br>
GEOS support: no<br>
<br>
GLw support: no<br>
<br>
JPEG support: yes<br>
<br>
LAPACK support: no<br>
<br>
Large File support (LFS): no<br>
<br>
Motif support: no<br>
<br>
MySQL support: no<br>
<br>
NLS support: no<br>
<br>
ODBC support: no<br>
<br>
OGR support: yes<br>
<br>
OpenGL support: yes<br>
<br>
PNG support: yes<br>
<br>
PostgreSQL support: no<br>
<br>
Python support: yes<br>
<br>
Readline support: no<br>
<br>
SQLite support: no<br>
<br>
Tcl/Tk support: yes<br>
<br>
wxWidgets support: yes<br>
<br>
TIFF support: yes<br>
<br>
X11 support: yes<br>
<br>
<br>
<br>
Then: make<br>
<br>
GRASS GIS compilation log<br>
<br>
-------------------------<br>
<br>
Started compilation: Wed Dec 28 12:22:59 CET 2011<br>
<br>
--<br>
<br>
Errors in:<br>
<br>
No errors detected.<br>
<br>
--<br>
<br>
Finished compilation: Wed Dec 28 12:24:11 CET 2011<br>
<br>
<br>
<br>
<br>
<br>
And su -c 'make install' that seems ok.<br>
<br>
<br>
<br>
However when I run: grass64 -wxpython I have the following error message:<br>
<br>
<br>
<br>
Cleaning up temporary files ...<br>
<br>
Starting GRASS ...<br>
<br>
ERROR: wxGUI requires wxPython. No module named wxversion<br>
<br>
Error in GUI startup. If necessary, please<br>
<br>
report this error to the GRASS developers.<br>
<br>
Switching to text mode now.<br>
<br>
Hit RETURN to continue...<br>
<br>
<br>
<br>
Where did I go wrong?<br>
<br>
<br>
<br>
Many thanks in advance.<br>
<br>
<br>
<br>
A. Clerici<br>
<br>
Parma University<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://lists.osgeo.org/pipermail/grass-user/attachments/20111228/d8363dca/attachment-0001.html" target="_blank">http://lists.osgeo.org/pipermail/grass-user/attachments/20111228/d8363dca/attachment-0001.html</a><br>
<br>
------------------------------<br>
<br>
Message: 3<br>
Date: Wed, 28 Dec 2011 13:16:29 +0100<br>
From: Martin Landa <<a href="mailto:landa.martin@gmail.com">landa.martin@gmail.com</a>><br>
Subject: Re: [GRASS-user] Compile and install 6.4.2RC2<br>
To: Aldo Clerici <<a href="mailto:aldo.clerici@unipr.it">aldo.clerici@unipr.it</a>><br>
Cc: <a href="mailto:grass-user@lists.osgeo.org">grass-user@lists.osgeo.org</a><br>
Message-ID:<br>
<<a href="mailto:CA%2BEi1Ocvg1hBsjQNt_bPA7ys2pzWiDWDBFRuuo3KD-WBXZg2Aw@mail.gmail.com">CA+Ei1Ocvg1hBsjQNt_bPA7ys2pzWiDWDBFRuuo3KD-WBXZg2Aw@mail.gmail.com</a>><br>
Content-Type: text/plain; charset=ISO-8859-1<br>
<br>
Hi,<br>
<br>
2011/12/28 Aldo Clerici <<a href="mailto:aldo.clerici@unipr.it">aldo.clerici@unipr.it</a>>:<br>
> ERROR: wxGUI requires wxPython. No module named wxversion<br>
><br>
> Error in GUI startup. If necessary, please<br>
><br>
> report this error to the GRASS developers.<br>
><br>
> Switching to text mode now.<br>
><br>
> Hit RETURN to continue...<br>
><br>
><br>
><br>
> Where did I go wrong?<br>
<br>
you need to install package which contains `wxversion`. On Debian<br>
GNU/Linux it's eg. `python-wxversion`.<br>
<br>
Martin<br>
<br>
--<br>
Martin Landa <landa.martin <a href="http://gmail.com" target="_blank">gmail.com</a>> * <a href="http://geo.fsv.cvut.cz/%7Elanda" target="_blank">http://geo.fsv.cvut.cz/~landa</a><br>
<br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
grass-user mailing list<br>
<a href="mailto:grass-user@lists.osgeo.org">grass-user@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/grass-user" target="_blank">http://lists.osgeo.org/mailman/listinfo/grass-user</a><br>
<br>
<br>
End of grass-user Digest, Vol 68, Issue 48<br>
******************************************<br>
</blockquote></div><br>