[GRASSLIST:9350] Re: v.in.ascii problems

[Roger Bivand]

On Wed, 7 Dec 2005, Jose Gomez-Dans wrote:

> Hi!
> Some months ago, I posted about v.in.ascii barfing on large input files due
> to massive memory consumption. Going through the bug reports and through my
> memory, I seem to recall that this had been solved (even valgrind was used
> for this). However, trying to read a file in (using 6.0, 6.0.1 and the
> latest binary CVS), the problem is still there. Has the problem really gone
> away?
> 

There is an on-going discussion about this on the GRASS development list.  
>From a simple test I ran last night, v.in.ascii -b (the -b flag is new in
GRASS 6.1 CVS) does not build topology, and this removes one of the two
humps in memory consumption. The other hump (> 200MB for a 1M point file
with a single attribute) was associated with writing the dbf file (the
file is 60MB), and is where things stick now. In addition, the -b flag
leaves the vector data set at level 1 topology (absent), and almost all
vector commands need level 2. 

I do now know whether the use of a different database driver than the
default would help. The dbf writing stage preceeds the topology building,
so the two memory-intensive humps are separate, with topology being a
little larger. Reading 1M points on a 1.5GHz P4 with topology took about 7
minutes, without about half that time.

If somebody has access to an external DBMS, it might be worth tracking 
memory use while writing to that connection. Details of the discussion on 
the GRASS5 list.

> Many thanks!
> 

-- 
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand at nhh.no