[GRASS-user] d.vect fails for large dataset

Wed Jun 14 23:53:43 PDT 2017

On Wed, Jun 14, 2017 at 9:23 AM, Žofie Cimburová <zoficimburova at gmail.com>
wrote:
>
> Dear all,
>
> I am importing a fairly large polygon dataset (10 086 419 land cover
polygons covering the area of Finland) from PostGIS into GRASS 7.2.1
(Linux) using v.import:
>
> v.import --o input=PG:dbname=gisdata layer=zofie_cimburova.clip_finland
output=clip_finland3 (see outputs of v.import, and v.info at the end of the
mail).
>
> The import finishes successfully, but when visualizing the layer (i.e.
Zoom to selected maps), I get a following error:
>
> Failed to run command 'd.vect map=clip_finland3 at u_zofie.cimburova
fcolor=200:200:200 color=black width=0 size=5 type=point,line,area,face
icon=basic/x'. Details: d.vect: spindex_rw.c:1362: rtree_get_node:
Assertion `t->nb[level][which].n.level == level' failed.

Apparently something went wrong with writing the spatial index to file.
Maybe some of those collapsed areas of zero size are causing problems for
the spatial index (they will also cause lots of other problems when trying
to analyse the map). You can try to import again with v.in.ogr using the
suggested snapping threshold of 1e-8, then increasing the threshold by
factor 10 until the warnings about zero-size areas disappear. This might
take quite some time. Most likely there will still be overlapping areas.

Markus M
>
> Do you by any chance know what the problem might be? I thought it is
caused by overlaps in the dataset, but when I import a small subset of the
original dataset, the error does not occur. Is the problem in the amount of
data? If so, how do I deal with it?
>
> Thanks a lot,
>
> Zofie
>
>
>
> v.info map=clip_finland3:
>
> (…)
>
>
|----------------------------------------------------------------------------|
>
> | Map format:      native
   |
>
>
|----------------------------------------------------------------------------|
>
> |   Type of map: vector (level: 2)
    |
>
> |
   |
>
> |   Number of points:       0               Number of centroids:
 12653604   |
>
> |   Number of lines:        0               Number of boundaries:
19187847   |
>
> |   Number of areas:        12728699        Number of islands:    8610980
   |
>
> |
   |
>
> |   Map is 3D:              No
    |
>
> |   Number of dblinks:      1
   |
>
> |
   |
>
> |   Projection: UTM (zone 33)
   |
>
> |
   |
>
> |               N:        7828443.03    S:   6602796.5606247
    |
>
> |               E:  1335791.79465162    W:         721051.43
    |
>
> |
   |
>
> |   Digitization threshold: 0
   |
>
> |   Comment:
    |
>
> |
   |
>
>
+----------------------------------------------------------------------------+
>
>
>
> v.import:
>
>
>
> Check if OGR layer <zofie_cimburova.clip_finland> contains polygons...
>
> Importing 10086419 features (OGR layer <zofie_cimburova.clip_finland>)...
>
> -----------------------------------------------------
>
> Registering primitives...
>
> 19371733 primitives registered
>
> 519601629 vertices registered
>
> Number of nodes: 12123276
>
> Number of primitives: 19371733
>
> Number of points: 0
>
> Number of lines: 0
>
> Number of boundaries: 19371733
>
> Number of centroids: 0
>
> Number of areas: -
>
> Number of isles: -
>
> -----------------------------------------------------
>
> Cleaning polygons
>
> -----------------------------------------------------
>
> Snapping boundaries (threshold = 1.000e-13)...
>
> Reading features...
>
> Snap vertices Pass 1: select points
>
> Snap vertices Pass 2: assign anchor vertices
>
> Snap vertices Pass 3: snap to assigned points
>
> -----------------------------------------------------
>
> Breaking polygons...
>
> Breaking polygons (pass 1: select break points)...
>
> Breaking polygons (pass 2: break at selected points)...
>
> -----------------------------------------------------
>
> Removing duplicates...
>
> -----------------------------------------------------
>
> Breaking boundaries...
>
> -----------------------------------------------------
>
> Removing duplicates...
>
> -----------------------------------------------------
>
> Cleaning boundaries at nodes...
>
> -----------------------------------------------------
>
> (...)
>
> -----------------------------------------------------
>
> Merging boundaries...
>
> -----------------------------------------------------
>
> Removing dangles...
>
> -----------------------------------------------------
>
> Building areas...
>
> WARNING: Area of size = 0.0 ignored
>
> (...)
>
> 12728699 areas built
>
> 8610980 isles built
>
> Attaching islands...
>
> Number of nodes: 15022766
>
> Number of primitives: 19187847
>
> Number of points: 0
>
> Number of lines: 0
>
> Number of boundaries: 19187847
>
> Number of centroids: 0
>
> Number of areas: 12728699
>
> Number of isles: 8610980
>
> WARNING: Number of incorrect boundaries: 52223
>
> WARNING: Vect_get_point_in_poly_isl(): collapsed area
>
> WARNING: Unable to calculate area centroid
>
> (...)
>
> -----------------------------------------------------
>
> Finding centroids for OGR layer <zofie_cimburova.clip_finland>...
>
> -----------------------------------------------------
>
> Writing centroids...
>
> WARNING: 2258413 areas represent more (overlapping) features, because
polygons overlap in input layer(s). Such areas are linked to more than 1
row in attribute table. The number of features for those areas is stored as
category in layer 2
>
> -----------------------------------------------------
>
> 11478002 input polygons
>
> Total area: 3.93306E+11 (12728699 areas)
>
> Overlapping area: 4.43546E+09 (2258413 areas)
>
> Area without category: 0.1304 (6448 areas)
>
> -----------------------------------------------------
>
> Copying features...
>
> Building topology for vector map <clip_finland3 at u_zofie.cimburova>...
>
> Registering primitives...
>
> 31841451 primitives registered
>
> 292562260 vertices registered
>
> Building areas...
>
> WARNING: Area of size = 0.0 ignored
>
> (...)
>
> 12728699 areas built
>
> 8610980 isles built
>
> Attaching islands...
>
> Attaching centroids...
>
> Number of nodes: 15022766
>
> Number of primitives: 31841451
>
> Number of points: 0
>
> Number of lines: 0
>
> Number of boundaries: 19187847
>
> Number of centroids: 12653604
>
> Number of areas: 12728699
>
> Number of isles: 8610980
>
> WARNING: Number of incorrect boundaries: 52223
>
> -----------------------------------------------------
>
> Some input polygons are overlapping each other.
>
> If overlapping is not desired, the data need to be cleaned.
>
> The input could be cleaned by snapping vertices to each other.
>
> Estimated range of snapping threshold: [1e-08, 1]
>
> Try to import again, snapping with at least 1e-08: 'snap=1e-08'
>
> Input <PG:dbname=gisdata> successfully imported without reprojection
>
> (Tue Jun 13 00:50:36 2017) Command finished (509 min 22 sec)
>
>
> _______________________________________________
> grass-user mailing list
> grass-user at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/grass-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20170615/6cd13715/attachment-0001.html>