[GRASS-dev] Re: [GRASS GIS] #1421: scalability of r.terraflow
GRASS GIS
trac at osgeo.org
Tue Aug 23 20:24:13 EDT 2011
#1421: scalability of r.terraflow
-------------------------------------+--------------------------------------
Reporter: dnewcomb | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.0.0
Component: Default | Version: svn-develbranch6
Keywords: r.terraflow large grids | Platform: Linux
Cpu: x86-64 |
-------------------------------------+--------------------------------------
Comment(by dnewcomb):
OK, I got it to work with the big grid.
for types.h
typedef short dimension_type; /* represent dimension of the grid */
static const dimension_type dimension_type_max=SHORT_MAX;
changed to :
typedef long dimension_type; /* represent dimension of the grid */
static const dimension_type dimension_type_max=LONG_MAX;
for 3scan.h:
line 127
assert(ae == AMI_ERROR_END_OF_STREAM);
changed to :
assert((off_t)ae == AMI_ERROR_END_OF_STREAM);
line 141
assert(ae == AMI_ERROR_END_OF_STREAM)
changed to:
assert((off_t)ae == AMI_ERROR_END_OF_STREAM)
output from command:
GRASS 7.0.svn (ncstpft_nad83):/data2/grass7_svn/grass_trunk/bin.x86_64
-unknown-linux-gnu > r.terraflow --overwrite elevation=nc_20ft_ncfpm
filled=nc_fill direction=nc_direct swatershed=nc_sink
accumulation=nc_flow_accum tci=nc_tci memory=8000
stream_dir=/data2/bareearth stats=/data2/bareearth/stats2.out
STREAM temporary files in /data2/bareearth (THESE INTERMEDIATE STREAMS
WILL NOT BE DELETED IN CASE OF ABNORMAL TERMINATION OF THE PROGRAM. TO
SAVE SPACE PLEASE DELETE THESE FILES MANUALLY!)
MFD flow direction
D8CUT=999999986991104.000000
Memory size: 7.81G (8388608000) bytes
Memory manager registering memory in MM_IGNORE_MEMORY_EXCEEDED mode.
total elements=6783000000, nodata elements=3291491362
largest temporary files:
FILL: 454.84G (488376000000) [-1806934592 elements, 72B each]
FLOW: 312.17G (335184829248) [3491508638 elements, 96B each]
Will need at least 909.67G (976752000000) space available in
/data2/bareearth
------------------------------
COMPUTING FLOW DIRECTIONS
classifying nodata (inner & boundary)
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7997.93MB
EMPQUEUEADAPTIVE: desired memory: 7997.93MB
sz_stream: 270400 buf_arity: 200 mm_overhead: 8666496 mm_avail:
8386435434.
EMPQUEUEADAPTIVE: memory overhead set to 8.26501MB
EMPQUEUEADAPTIVE: pqsize set to 1047221117
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7997.15MB
EMPQUEUEADAPTIVE: desired memory: 7997.15MB
sz_stream: 270400 buf_arity: 200 mm_overhead: 8666496 mm_avail:
8385624130.
EMPQUEUEADAPTIVE: memory overhead set to 8.26501MB
EMPQUEUEADAPTIVE: pqsize set to 1047119704
assigning preliminary directions
finding flat areas (plateaus and depressions)
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7997.41MB
EMPQUEUEADAPTIVE: desired memory: 7997.41MB
sz_stream: 270400 buf_arity: 200 mm_overhead: 8666496 mm_avail:
8385894538.
EMPQUEUEADAPTIVE: memory overhead set to 8.26501MB
EMPQUEUEADAPTIVE: pqsize set to 1047153505
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7996.64MB
EMPQUEUEADAPTIVE: desired memory: 7996.64MB
sz_stream: 270400 buf_arity: 200 mm_overhead: 8666496 mm_avail:
8385083234.
EMPQUEUEADAPTIVE: memory overhead set to 8.26501MB
EMPQUEUEADAPTIVE: pqsize set to 1047052092
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7995.86MB
EMPQUEUEADAPTIVE: desired memory: 7995.86MB
sz_stream: 270400 buf_arity: 200 mm_overhead: 8666496 mm_avail:
8384271930.
EMPQUEUEADAPTIVE: memory overhead set to 8.26501MB
EMPQUEUEADAPTIVE: pqsize set to 1046950679
assigning directions on plateaus
generating watersheds and watershed graph
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7998.96MB
EMPQUEUEADAPTIVE: desired memory: 7998.96MB
sz_stream: 270424 buf_arity: 200 mm_overhead: 8705664 mm_avail:
8387517074.
EMPQUEUEADAPTIVE: memory overhead set to 8.30237MB
EMPQUEUEADAPTIVE: pqsize set to 261837856
flooding depressions
available memory: 7999MB (8387787594B)
UnionFind::makeSet: reallocate double 2000
UnionFind::makeSet: reallocate double 4000
UnionFind::makeSet: reallocate double 8000
UnionFind::makeSet: reallocate double 16000
UnionFind::makeSet: reallocate double 32000
UnionFind::makeSet: reallocate double 64000
UnionFind::makeSet: reallocate double 128000
UnionFind::makeSet: reallocate double 256000
UnionFind::makeSet: reallocate double 512000
UnionFind::makeSet: reallocate double 1024000
UnionFind::makeSet: reallocate double 2048000
UnionFind::makeSet: reallocate double 4096000
UnionFind::makeSet: reallocate double 8192000
UnionFind::makeSet: reallocate double 16384000
UnionFind::makeSet: reallocate double 32768000
warning: watershed 1 (R=1) not done
warning: watershed 31667557 (R=31688834) not done
warning: watershed 31667558 (R=31688834) not done
warning: watershed 31674901 (R=31688834) not done
warning: watershed 31676231 (R=31688834) not done
warning: watershed 31688834 (R=31688834) not done
------------------------------
REASSIGNING DIRECTIONS
finding flat areas (plateaus and depressions)
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7997.15MB
EMPQUEUEADAPTIVE: desired memory: 7997.15MB
sz_stream: 270400 buf_arity: 200 mm_overhead: 8666496 mm_avail:
8385624138.
EMPQUEUEADAPTIVE: memory overhead set to 8.26501MB
EMPQUEUEADAPTIVE: pqsize set to 1047119705
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7996.38MB
EMPQUEUEADAPTIVE: desired memory: 7996.38MB
sz_stream: 270400 buf_arity: 200 mm_overhead: 8666496 mm_avail:
8384812834.
EMPQUEUEADAPTIVE: memory overhead set to 8.26501MB
EMPQUEUEADAPTIVE: pqsize set to 1047018292
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7995.61MB
EMPQUEUEADAPTIVE: desired memory: 7995.61MB
sz_stream: 270400 buf_arity: 200 mm_overhead: 8666496 mm_avail:
8384001530.
EMPQUEUEADAPTIVE: memory overhead set to 8.26501MB
EMPQUEUEADAPTIVE: pqsize set to 1046916879
EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7994.83MB
EMPQUEUEADAPTIVE: desired memory: 7994.83MB
sz_stream: 270400 buf_arity: 200 mm_overhead: 8666496 mm_avail:
8383190226.
EMPQUEUEADAPTIVE: memory overhead set to 8.26501MB
EMPQUEUEADAPTIVE: pqsize set to 1046815466
assigning directions on plateaus
creating flowStream: [AMI_STREAM /data2/bareearth/flowStream 0]
compute flow directions done.
100%
100%
100%
------------------------------
COMPUTING FLOW ACCUMULATION
creating sweep stream from fill output stream
sorting sweep stream
sweeping: EMPQUEUEADAPTIVE: starting in-memory pqueue
EMPQUEUEADAPTIVE: available memory: 7999.73MB
EMPQUEUEADAPTIVE: desired memory: 7999.73MB
sz_stream: 270424 buf_arity: 200 mm_overhead: 8705664 mm_avail:
8388328213.
EMPQUEUEADAPTIVE: memory overhead set to 8.30237MB
EMPQUEUEADAPTIVE: pqsize set to 261863204
100%
sorting sweep output stream
100%
r.terraflow complete.
--
Ticket URL: <http://trac.osgeo.org/grass/ticket/1421#comment:1>
GRASS GIS <http://grass.osgeo.org>
More information about the grass-dev
mailing list