[GRASS-dev] large vector problems

[Markus Metz]

Wouter Boasson wrote:
> Dear Markus, Markus and Jens,
>
> My MacBook survived the burn-in test :-) After 29 hours under full load,
> alternating processor and disk access limited, I got my mega-vector file
> cleaned. Thank you for the support and suggestions to solve my problems
> with the large vector cleaning operation!
>   
Glad to hear that it worked in the end!
> Although the dataset was cleaned, files of this size are virtually
> impossible to handle, especially as standard querying, extract and overlay
> operations with raster datasets simply take too much time. 
There are ways to improve both speed and memory consumption. There are 
hints in the source code and I have some ideas, but this is no easy 
task. The grass vector model is complex, changes need a lot of testing 
before they can be applied. And there are not that many developers 
working on the core grass vector libraries... This will only happen in 
grass7, I guess. Hopefully sometime this year...
> I'll post a few related issues with mega files that makes working with them
> very difficult. I'll post them as (separate) enhancements on trac, as they
> are in my opinion of major importance:
> -selecting a large vector map from a dropdown box in the wxPython GUI takes
> a long time
> -renaming this vector took 25 minutes (PostgreSQL access!)
> -v.extract is also incredibly slow
> -removing a vector file with an unreachable PostgreSQL database link does
> not work, not even in force mode
> -v.what consuming several GB of RAM only for querying a large vector map??
>   
Some of the above operations could be improved, but it will take some time.
> -v.rast.stats suffers from setting masks, extracting polygons and querying,
> not usefull anymore for vector files this size, this is a particular slow
> operations
>   
Try the example script in the help page of r.univar.zonal, available in 
the grass-addons:
http://grass.osgeo.org/wiki/GRASS_AddOns#r.univar.zonal
It should be easy to modify it to your needs, it does something very 
similar to v.rast.stats, only faster.

Best regards,

Markus M