[GRASSLIST:7189] Re: v.in.ascii out of memory

[Andrew Danner]

On Thu, 2005-06-16 at 15:42 +1200, Hamish wrote:
> > I'm trying to import a somewhat large file of 24.8 million LIDAR
> > points (866MB) into GRASS 6.1 (11 Jun CVS) using v.in.ascii
> 
> Status: (all due to memory limitations [ie leaks])
>   GRASS 6.0.0's v.in.ascii   ~ 100k points
>   GRASS 6.1-cvs v.in.ascii   ~ 3M points (you)
>
> > I'm using
> > 
> > v.in.ascii -z input=strip_09.txt output=g6Npts fs=, z=3
> 
> try with -t too. No need to create a table if you have no attributes!
> 

-t saves maybe a few KB. Good to know, but not the main culprit

>  
> > and I get the following:
> > 
> > Building topology ...
> > Registering lines: ERROR: G_realloc: out of memory
> > 
> > There shouldn't be any lines to register. This is just a list of
> > points. Any ideas?
> 
> I've seen the same thing. Not sure if it has to know everything to 
> build topology or if it is a memory leak. I think it's a leak.
> 
> I did some tests with valgrind back in March. Radim plugged the biggest 
> leak, but I think "Registering Lines.." still has one too.
> 
> see this thread:
>  http://article.gmane.org/gmane.comp.gis.grass.devel/7212/
> 
>  
> 
> > At three million points, I noticed that I'm using over 1GB of memory.
> > Is it trying to build the entire topology in memory? I don't need to
> > build any topology for a set of points. Is there a way to turn this
> > off? 
> >
> 
> a) Try adding more swap.
> b) We need to fix the leak. (more valgrind)
> 

I'm not running out of swap. In fact, isn't it the case that the 32-bit
Linux machines can address more the 4GB of address space per process (or
perhaps per thread)?, so at some point, more swap won't work. Unless the
topology building is doing something where each feature, reading in a
list of points should have memory requirements that are independent on
the input size and I should only be limited by disk space. 

-Andy