[GRASS-dev] v.in.ascii memory errors again

[Hamish]

Andrew Danner wrote:

> I'm having problems importing huge lidar point sets using v.in.ascii.
> I thought this issue was resolved with the -b flag, but v.in.ascii is
> consuming all the memory even in the initial scan of the data (before
> building the topology, which should be skipped with the -b flag)
> 
> My data set is comma separated x,y,z points
> 
> v.in.ascii -ztb input=BasinPoints.txt output=NeuseBasinPts fs="," z=3
> 
> Sample data: 
> 
> 1939340.84,825793.89,657.22
> 1939071.95,825987.78,660.22
> 1939035.52,826013.97,662.46
> 1938762.45,826210.15,686.28
> 1938744.05,826223.34,688.57
> 1938707.4,826249.58,694.1
> 1938689.21,826262.62,696.55
> 1938670.91,826275.77,698.07
> 1938616.48,826314.99,699.31
> 1938598.36,826328.09,698.58
> 
> I have over 300 million such records and the input file is over 11GB. 
> 
> v.in.ascii runs out of memory and crashes during points_analyse in
> v.in.ascii/points.c


Hi,

you'll probably want to use the new r.in.xyz module for anything more
than 3 million points. I think v.surf.rst is the only module which can
do something useful with vector points without topology.

http://grass.ibiblio.org/grass61/manuals/html61_user/r.in.xyz.html
http://hamish.bowman.googlepages.com/grassfiles#xyz

v.in.ascii (without -b) has finite memory needs due to topological support.
Search the mailing list archives for many comments on "v.in.ascii memory
leak" by Radim, Helena, and myself on the subject. Here is some valgrind
analysis I did on it at the time:

http://bambi.otago.ac.nz/hamish/grass/memleak/v.in.ascii/

If you can find a way to lessen the memory footprint, then great!
Same for large file and 64bit support fixes.

> I can now see why the free was originally moved outside the loop
> to fix lat/long problems: because tokens[i] is redirected to a
> different buffer in the LL case. This seems problematic and a possible
> source of memory leaks.

but LL parsing support was only added relatively recently? need to check
the CVS log. rev 1.9:
http://freegis.org/cgi-bin/viewcvs.cgi/grass6/vector/v.in.ascii/points.c

so not a "core" unfixable part of the code?

Radim said that freeing memory was slow. Maybe free a chunk of memory
every 50000th point or so?



Hamish