[GRASS-user] v.in.ascii : ERROR: write sidx

[Hamish]

Yann wrote:

> I tried v.in.ascii from the Layer manager (i use sqlite driver) :
>
>(Thu Sep 26 23:58:29 2013)                                                      
>v.in.ascii
-z --overwrite --verbose 
input=/media/yann/DATA_SIG/3D_photo/menhir2_pc_88M.csv 
output=PTS_menhir2_pc_88M separator=  z=3
>WARNING: Vector map <PTS_menhir2_pc_88M> already exists and will be overwritten
>Using native format
>Scanning input for column types...
>Maximum input row length: 30
>Number of columns: 3
>Importing points...
>Building topology for vector map <PTS_menhir2_pc_88M at menhir2>...
>Registering primitives...
>88509169 primitives registered
>88509169 vertices registered
>Building areas...
>0 areas built
>0 isles built
>Attaching islands...
>Attaching centroids...
>Topology was built
>Number of nodes: 0
>Number of primitives: 88509169
>Number of points: 88509169
>Number of lines: 0
>Number of boundaries: 0
>Number of centroids: 0
>Number of areas: 0
>Number of isles: 0
>ERROR: write sidx: wrong node position in file
>(Fri Sep 27 00:44:28 2013) Command finished (45 min 59 sec)
>
> I have some disk space, but i looked at my system monitor and my RAM
> went to SWAP (i have only 16 Go RAM, 15 Go SWAP) The swap don't seems
> to have been totally used (but at least half) and i have not in front
> of my PC when the error occurred.
>
> I will try this again with more RAM.

Ok, 88 million points and running 64bit Linux. Since 3D points
and no additional data columns columns I think it will default
to not creating a database, so you don't have to worry about the
-t flag or the memory needed for that, just the finite amount of
RAM per point for topology. Maybe 64GB RAM will be enough for 88M,
perhaps more, but it will need a lot to build topology for all those
points.

May I ask what process you'd like to do with the data? Perhaps
r.in.xyz + r.to.vect helps reduce the number of points to something
more manageable. The v.in.ascii '-r' flag can be helpful for importing
only those points within the current computational region of interest,
and if importing without building topology (the '-b' flag you tried
before) many of the LiDAR modules and v.surf.rst have been modified
to work without needing it.
If it is some other vector module maybe we can modify it to load data
without topology too.
Is it sparse x,y data or gridded?


regards,
Hamish