[GRASS-user] Large vector files

[Moritz Lennert]

On Fri, October 6, 2006 22:53, Glynn Clements wrote:
>
> Jonathan Greenberg wrote:
>
>> Unfortunately, I was hoping to work in a vector environment with the
>> data --
>> I'm sure I could think up raster analogs to the analyses I'm trying to
>> do
>> right now, but as Michael pointed out earlier, this is a problem that
>> does
>> need to be solved -- Lidar, in particular, is getting more popular, and
>> software support remains primitive -- while my problem is not a Lidar
>> one,
>> it still has the same underlying issue -- I need to be able to create
>> and
>> manipulate massive vector files.
>>
>> I am hearing a lot of suggestions about using things like PostGIS and
>> PostGRESQL here and elsewhere, but I am a total novice to this -- is
>> there a
>> "dummy's guide" to working with these DB instead of shapefiles?  I'd
>> like to
>> be able to use all of the various GRASS vector commands on a "large
>> vector"
>> (what would the format be called?) that already exist -- I'm noticing
>> there's a lot more setup involved in getting a DB running, and the
>> process
>> has yet to be streamlined.  Is it possible to simply substitute some
>> postgres driven vector DB for a GRASS vector in the GRASS algorithms, or
>> do
>> the v.[whatever] algorithms need to be reworked to support this?
>
> You can use any supported database system to store vectors. Use
> db.connect to select which database to use.

No, db.connect tells you which database to use for the link between your
GRASS vector maps and attribute tables, not where to store the actual
vector geometries, which are normally stored internally.

>From what I have gathered from your mails your problem does not seem to be
the size of the attribute table, nor necessarily a large file problem, but
rather the size of the geometry file (number of points). Thus the
v.in.ascii error. The memory problem here comes from the building of
topology where, as Hamish explained, there is a memory overhead which
becomes significant with large numbers of features.

If you want to use another vector format than the internal GRASS format,
you can try to use v.external on a shape file or a PostGIS table. But I
don't know if that solves the memory issues, if you need topology for your
operations.

PostGIS also contains a series of vector operations, so you might be able
to do what you want to do without going through GRASS.

Maybe you could tell us what exactly you want to do...

>
> I'm not particularly familiar with the vector side of GRASS, but it
> appears that whether or not the map includes topology information has
> a significant effect upon the maximum size. Maps which lack topology
> information are less likely to cause problems than those which have
> it.
>

The latest discussions on this went in the direction of reenabling spatial
index on file, instead of in memory. This should solve the topology issue,
but it still needs to be done (Radim gave a rough roadmap of how to do
it).

None of what I say above answers your question about large file support,
though...

Moritz