[GRASS-user] memory requirement of v.net.allpairs module

[Markus Metz]

On Sun, Dec 25, 2011 at 5:04 PM, Pankaj Kr Sharma <pkscwc at gmail.com> wrote:
> Dear Developers and users,
>
> On a small test region (11000x11000 cells), with a threshold of 1000sqkm,
> r.stream.extract resulted in a stream vector network of 190 points and
> lines. The v.net.allpairs generated a table of distances between each point
> within 10 seconds on my home machine (only 4gb ram).  I thought that I got
> my solution for finding sub-basins of a very large area. (Please refer to my
> earlier posts in this regard.)
>
> On my work setup, (better machine, 16gb RAM , six cores), I followed this
> routine:
> 1. r.grow by one cell  on area of interest. (It's an important step. Thanks
> to Markus Metz for this tip.) Time taken by the module: approximately 2
> hours.
>
> 2. r.stream.extract on my area of interest.(42000*42000 cells) with 100
> sq.km. as threshold (12345 in number of cells.)

Above you said that yourtest region was 11000x11000 cells, not
42000x42000 cells. Both r.grow and r.stream.extract will be
considerably faster if you adjust your computational region to your
test region.

>     Please note that the r.stream.extract module's vector output was a
> network of 1,45,000 (approx) points and lines.
> Time taken by module: approximately 24 hours.
>
> 3. Then I ran "v.net.allpairs" on this network. Now, v.net.allpairs is still
> running. Approximately, one week has elapsed.
> After three days, a message appeared on the console that says "writing data
> into the table". After four days of this message, the progress bar is still
> without any colour. ( Which I consider as progress being less than one
> percent.)
>
> The other thing which I noticed was the memory consumption. The module has
> consumed 15.5 gb ram out of 16gb, thereafter it consumed 166gb of swap space
> out of 500gb swap space available.( Earlier, I had kept complete one disk of
> 1 tb for swap.)

This sounds like bad news. v.net.allpairs keeps all <from> categories,
all <to> categories, and all distances from <from> to <to> in memory.
For 145000 network nodes, this would be about 320 GB. Assuming that
half of the features are lines and half of the features are points,
v.net.allpairs would require about 160GB of memory, plus some overhead
to handle the vector if you did not set the environment variable
GRASS_VECTOR_LOWMEM. Granted that you did not run v.net.allpairs with
the -a flag.

Why did you use v.net.allpairs at all on the stream network?

Markus M

>
> Some of the answers for this response of the module, I have myself.
>
> Like, the module has more calculations to do.
> (145000*145000)/(190*190)~582410.
> Still, as per my calculation of time required 582410*10 seconds/(24*60*60)~
> 6.75 days
>
> And, I should have my postgres table ready for further calculations.
> But, still there is no sign of any progress.
> Is the time and memory being consumed by the module as per design and ok.
>
> I can definitely sub-divide into basins and work further.
>
>
> _______________________________________________
> grass-user mailing list
> grass-user at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user
>