[GRASS-user] Large vector files

[Jonathan Greenberg]

I wonder (and I'm thinking out loud here) if there are ways to "tile" vector
processes in an analogous (if not algorithmic) way to how we deal with
massive raster datasets?  Are the issues I'm running into fundamentally
something with older file formats, operating system/file system limitations,
algorithmic maturity, or some mixture of all of these things?  As you
pointed out, the Lidar community seems to have the most pressing need for
these issues to get sorted out -- however as GIS analyses get more advanced
and require more data, I'm guessing the average user may run into this as
well.

On a related note, apparently ESRI may be releasing a new version of their
geodatabase format to get around some of the filesize issues in their 9.2
release (the beta apparently has this functionality).  No word on whether it
a) works or b) has algorithmic advances to deal with these DB...

--j


On 10/5/06 4:16 PM, "Hamish" <hamish_nospam at yahoo.com> wrote:

> Jonathan Greenberg wrote:
>> 
>> Case in point: I just got this error on a v.in.ascii import of a
>> ~200mb csv file with points:
>> 
>> G_realloc: out of memory (I have 4gb RAM and plenty of swap space, and
>> the program never hit that limit anyway).
> 
> 
> The vector format has a small but finite memory overhead for each
> feature which makes more than several million data points impractical.
> 
> To get around this v.in.ascii (and a couple of other modules) let you
> load in vector data without building topology.  (v.in.ascii -b -t)
> 
> Then it's unknown how many points you can load, but it's a lot.
> 
> Without topology, about the only thing you can do with the data is run
> it through v.surf.rst.
> 
> 
> For multi-gigabyte x,y,z datasets (or x,y,f(x,y) just as well), you can
> use r.in.xyz to bin it directly into a raster map.
> 
> see:
>   http://grass.ibiblio.org/grass63/manuals/html63_user/r.in.xyz.html
>   http://hamish.bowman.googlepages.com/grassfiles#xyz
> 
> 
> with regard to the vector library and LFS support, I think you can
> expect some "first user" problems, Radim commented on this some time ago
> in the mailing lists, have to search there for a better answer.
> 
> 
> Hamish


-- 
Jonathan A. Greenberg, PhD
NRC Research Associate
NASA Ames Research Center
MS 242-4
Moffett Field, CA 94035-1000
Office: 650-604-5896
Cell: 415-794-5043
AIM: jgrn307
MSN: jgrn307 at hotmail.com