[GRASSLIST:7177] Re: v.in.ascii out of memory

[Hamish]

> I'm trying to import a somewhat large file of 24.8 million LIDAR
> points (866MB) into GRASS 6.1 (11 Jun CVS) using v.in.ascii

Status: (all due to memory limitations [ie leaks])
  GRASS 6.0.0's v.in.ascii   ~ 100k points
  GRASS 6.1-cvs v.in.ascii   ~ 3M points (you)


> File format is just a list of easting, northing, and elevation in NC
> state-plane. 
> 
> 1939340.8400,825793.8900,657.2200
> 1939071.9500,825987.7800,660.2200
> 1939035.5200,826013.9700,662.4600
> 1938762.4500,826210.1500,686.2800
> 
> 
> I'm using
> 
> v.in.ascii -z input=strip_09.txt output=g6Npts fs=, z=3

try with -t too. No need to create a table if you have no attributes!

 
> and I get the following:
> 
> Maximum input row length: 34
> Maximum number of columns: 3
> Minimum number of columns: 3
> 
> Building topology ...
> Registering lines: ERROR: G_realloc: out of memory
> 
> There shouldn't be any lines to register. This is just a list of
> points. Any ideas?

I've seen the same thing. Not sure if it has to know everything to 
build topology or if it is a memory leak. I think it's a leak.

I did some tests with valgrind back in March. Radim plugged the biggest 
leak, but I think "Registering Lines.." still has one too.

see this thread:
 http://article.gmane.org/gmane.comp.gis.grass.devel/7212/

 
> I have imported close to 500 million points in GRASS 5.4 using
> s.in.ascii, but I don't know why things are crashing in the new vector
> format. 

GRASS 5.4 stored sites in a simple ASCII file-- the limit was whatever
your filesystem/OS could handle. Vector points in GRASS 6 is a 100% 
different implementation.

 
> I have tested v.in.ascii upto 3 million points. 
> 
> I'm running on a Fedora Core 3 Linux with a 2.6 kernel, 1GB of
> physical memory and 8GB of swap. 

Note you can add more swap memory to Linux on the fly, see:

 http://grass.itc.it/pipermail/grassuser/2002-February/006072.html


> At three million points, I noticed that I'm using over 1GB of memory.
> Is it trying to build the entire topology in memory? I don't need to
> build any topology for a set of points. Is there a way to turn this
> off? 
>
> In 5.4 I was developing a scalable version of s.surf.rst that could
> interpolate surfaces on over 500 million points, but now I can't even
> import that many points into a vector/site format. 
> 
> Comments/Suggestions?

a) Try adding more swap.
b) We need to fix the leak. (more valgrind)


let us know how you get on.


Hamish