[GRASS-dev] v.in.ascii memory errors again

[Helena Mitasova]

Hamish,

thanks for the answer, but as Andy has mentioned, he is using v.in.ascii 
-b already and v.surf.rst (his version).
Andy is not the only one who needs v.in.ascii -b to behave at least the 
same way as it did before the LL
related change - I just emailed other users who were asking about it 
that the v.in.ascii -b, v.surf.rst pipeline for processing large point
data sets works and it apparently is broken now.

Helena

Hamish wrote:
> Andrew Danner wrote:
>
>   
>> I'm having problems importing huge lidar point sets using v.in.ascii.
>> I thought this issue was resolved with the -b flag, but v.in.ascii is
>> consuming all the memory even in the initial scan of the data (before
>> building the topology, which should be skipped with the -b flag)
>>
>> My data set is comma separated x,y,z points
>>
>> v.in.ascii -ztb input=BasinPoints.txt output=NeuseBasinPts fs="," z=3
>>
>> Sample data: 
>>
>> 1939340.84,825793.89,657.22
>> 1939071.95,825987.78,660.22
>> 1939035.52,826013.97,662.46
>> 1938762.45,826210.15,686.28
>> 1938744.05,826223.34,688.57
>> 1938707.4,826249.58,694.1
>> 1938689.21,826262.62,696.55
>> 1938670.91,826275.77,698.07
>> 1938616.48,826314.99,699.31
>> 1938598.36,826328.09,698.58
>>
>> I have over 300 million such records and the input file is over 11GB. 
>>
>> v.in.ascii runs out of memory and crashes during points_analyse in
>> v.in.ascii/points.c
>>     
>
>
> Hi,
>
> you'll probably want to use the new r.in.xyz module for anything more
> than 3 million points. I think v.surf.rst is the only module which can
> do something useful with vector points without topology.
>
> http://grass.ibiblio.org/grass61/manuals/html61_user/r.in.xyz.html
> http://hamish.bowman.googlepages.com/grassfiles#xyz
>
> v.in.ascii (without -b) has finite memory needs due to topological support.
> Search the mailing list archives for many comments on "v.in.ascii memory
> leak" by Radim, Helena, and myself on the subject. Here is some valgrind
> analysis I did on it at the time:
>
> http://bambi.otago.ac.nz/hamish/grass/memleak/v.in.ascii/
>
> If you can find a way to lessen the memory footprint, then great!
> Same for large file and 64bit support fixes.
>
>   
>> I can now see why the free was originally moved outside the loop
>> to fix lat/long problems: because tokens[i] is redirected to a
>> different buffer in the LL case. This seems problematic and a possible
>> source of memory leaks.
>>     
>
> but LL parsing support was only added relatively recently? need to check
> the CVS log. rev 1.9:
> http://freegis.org/cgi-bin/viewcvs.cgi/grass6/vector/v.in.ascii/points.c
>
> so not a "core" unfixable part of the code?
>
> Radim said that freeing memory was slow. Maybe free a chunk of memory
> every 50000th point or so?
>
>
>
> Hamish
>
> _______________________________________________
> grass-dev mailing list
> grass-dev at grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass-dev
>   


-- 
Helena Mitasova
Department of Marine, Earth and Atmospheric Sciences
North Carolina State University
1125 Jordan Hall
NCSU Box 8208
Raleigh, NC 27695-8208
http://skagit.meas.ncsu.edu/~helena/

email: hmitaso at unity.ncsu.edu
ph: 919-513-1327 (no voicemail)
fax 919 515-7802