[GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless

[Roger Bivand]

On Tue, 6 Dec 2005, Maciek Sieczka wrote:

> On wto, 2005-12-06 at 22:32 +1300, Hamish wrote:
> > well what do you know, a first step just 8 day ago:
> > 
> > http://grass.itc.it/pipermail/grass-commit/2005-November/019366.html
> 
> Cool. Can anybody say how many points without topology I can import
> using this modified v.in.ascii?
> 
> Maciek

Timings on a 1.5GHz P4 (1GB), 2D points and single int value

500 points topo < 1 second
500 points notopo < 1 second
50K points topo 17 seconds
50K points notopo 10 seconds
1M points topo 403 seconds - dbf process > 200MB, v.in.ascii > 300MB
1M points notopo 216 seconds - dbf process > 200MB

so -b saves about half the time for 1M points, but the dbf process is 
still large and with more points will lead to swapping - this is writing 
out the data to the dbf file. Maybe a real DBMS would relieve this, I 
don't know. So even with -b, you are still constrained by memory, and 
simple modules like v.info can't handle the -b case:

GRASS 6.1.cvs (tull):~/tmp/gpts > v.info gptsb
ERROR: Cannot open old vector gptsb at rsb on level 2
GRASS 6.1.cvs (tull):~/tmp/gpts > v.info gpts
 +----------------------------------------------------------------------------+
 | Layer:    gpts                           Organization:                     |
 | Mapset:   rsb                            Source Date:                      |
 | Location: tull                           Name of creator:                  |
 | Database: /home/rsb/topics/grassdata                                       |
 | Title:                                                                     |
 | Map Scale:  1:1                                                            |
 | Map format: native                                                         |
 |----------------------------------------------------------------------------|
 |   Type of Map:  Vector (level: 2)                                          |
 |                                                                            |
 |   Number of points:       1000000         Number of areas:      0          |
 |   Number of lines:        0               Number of islands:    0          |
 |   Number of boundaries:   0               Number of faces:      0          |
 |   Number of centroids:    0               Number of kernels:    0          |
 |                                                                            |
 |   Map is 3D:              0                                                |
 |   Number of dblinks:      1                                                |
 |                                                                            |
 |   Projection: UTM (zone 0)                                                 |
 |            N: 56.000        S: 46.000                                      |
 |            E: 25.000         W: 5.000                                      |
 |            B: 0.000         T: 0.000                                       |
 |                                                                            |
 |   Digitize threshold: 0.00000                                              |
 |   Comments:                                                                |
 |                                                                            |
 +----------------------------------------------------------------------------+

Data generated in R:

set.seed(051206)
v <- rpois(1000000, 6)
x <- runif(1000000, 5, 25)
y <- runif(1000000, 46, 56)
cat(file="gpts.txt", paste(x, "|", y, "|", v, "\n", sep=""))
cat(file="gptsS.txt", paste(x[1:50000], "|", y[1:50000], "|", v[1:50000], 
"\n", sep=""))
cat(file="gptsXXS.txt", paste(x[1:500], "|", y[1:500], "|", v[1:500], 
"\n", sep=""))

> 
> 
> --------------------
> W polskim Internecie s± setki milionów stron. My przekazujemy Tobie tylko najlepsze z nich!
> http://katalog.epf.pl/
> 

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no