[GRASS5] [bug #3354] (grass) v.in.ascii crashing on large inputs

Request Tracker grass-bugs at intevation.de
Thu Jun 16 11:58:03 EDT 2005


this bug's URL: http://intevation.de/rt/webrt?serial_num=3354
-------------------------------------------------------------------------

Subject: v.in.ascii crashing on large inputs

Platform: GNU/Linux/i386
grass obtained from: CVS
grass binary for platform: Compiled from Sources
GRASS Version: 6.1 CVS 11 June

A followup to bug 2903. I cannot import more than about three or four million LIDAR points using v.in.ascii. It crashes with an out of memory error. 

From: 	Hamish
In reply to: Andrew Danner
Re: [GRASSLIST:7168] v.in.ascii out of memory
Date: 	Thu, 16 Jun 2005 15:42:02 +1200  (05:42 CEST)

> I'm trying to import a somewhat large file of 24.8 million LIDAR
> points (866MB) into GRASS 6.1 (11 Jun CVS) using v.in.ascii

Status: (all due to memory limitations [ie leaks])
  GRASS 6.0.0's v.in.ascii   ~ 100k points
  GRASS 6.1-cvs v.in.ascii   ~ 3M points (you)


> File format is just a list of easting, northing, and elevation in NC
> state-plane. 
> 
> 1939340.8400,825793.8900,657.2200
> 1939071.9500,825987.7800,660.2200
> 1939035.5200,826013.9700,662.4600
> 1938762.4500,826210.1500,686.2800
> 
> 
> I'm using
> 
> v.in.ascii -z input=strip_09.txt output=g6Npts fs=, z=3

try with -t too. No need to create a table if you have no attributes!

*This only offers a small improvement of < 100000 points
 
> and I get the following:
> 
> Maximum input row length: 34
> Maximum number of columns: 3
> Minimum number of columns: 3
> 
> Building topology ...
> Registering lines: ERROR: G_realloc: out of memory
> 
> There shouldn't be any lines to register. This is just a list of
> points. Any ideas?

I've seen the same thing. Not sure if it has to know everything to 
build topology or if it is a memory leak. I think it's a leak.

I did some tests with valgrind back in March. Radim plugged the biggest 
leak, but I think "Registering Lines.." still has one too.

see this thread:
 http://article.gmane.org/gmane.comp.gis.grass.devel/7212/


> At three million points, I noticed that I'm using over 1GB of memory.
> Is it trying to build the entire topology in memory? I don't need to
> build any topology for a set of points. Is there a way to turn this
> off? 
>
> In 5.4 I was developing a scalable version of s.surf.rst that could
> interpolate surfaces on over 500 million points, but now I can't even
> import that many points into a vector/site format. 
> 
> Comments/Suggestions?

a) Try adding more swap.
b) We need to fix the leak. (more valgrind)


*I have 8GB of swap. There may be some Linux 2.6 oom-killer issues, and I will try to fix this, but reading in just a list of points should not require over 1GB of memory. There is no topology to build. 



-------------------------------------------- Managed by Request Tracker




More information about the grass-dev mailing list