[GRASS-user] importing big vector data

[Martin Landa]

Hi all,

I was playing with big vector data and trying import it into GRASS
native format. I used 'buildings' layer from OSM provided in Esri
Shapefile for region of the Czech Republic [1].

This layer has almost 3.5 milions of polygons.

$ ogrinfo buildings.shp -al | grep 'Count:'
Feature Count: 3464029

I used PC with 16 CPUs (Intel Xeon @ 2.53GHz) a 48GB RAM with 64bit
Debian stable. GRASS compiled with large-file support.

1) $ v.in.ogr dsn=buildings.shp

-> after 30min failed with "Unexpected error. Aborted", memory usage peak 4.1GB

2) $ v.in.ogr -c dsn=buildings.shp

-> 20min with 3.0GB memory peak

The result was not topologically clean, but import procedure finished.

3) $ GRASS_VECTOR_LOWMEM=1 v.in.ogr dsn=buildings.shp

-> after 110min failed with "ERROR: G_calloc: unable to allocate 101 *
4 bytes of memory at allocation.c:81" (when Finding centroids for OGR
layer <buildings>...), memory peak 3.6GB

4) shp->postgis + v.external (simple features)

$ shp2pgsql -I -D -S buildings.shp | psql pgis_db

in 2min!

$ v.external dsn=PG:dbname=pgis_db layer=buildings

-> after 9min failed with "G_realloc: unable to allocate 40800000
bytes of memory at cindex.c:113"

GRASS_VECTOR_LOWMEM=1 v.external dsn=PG:dbname=pgis_db layer=buildings

-> it took 17min with memory peak 2.2GB, link was created!

I tried to render such data

$ d.mon cairo output=/tmp/buildings.png

$ g.region vect=buildings

$ d.vect buildings

but failed in 30sec with

ERROR: G_realloc: unable to allocate 48000 bytes of memory at list.c:46

Any comments, suggesting are highly welcomed...

Thanks, Martin

[1] http://download.geofabrik.de/europe/czech-republic-latest.shp.zip

-- 
Martin Landa * http://geo.fsv.cvut.cz/gwiki/Landa