[GRASS-dev] [GRASS GIS] #494: break in importing and cleaning very large vector datasets

[GRASS GIS]

#494: break in importing and cleaning very large vector datasets
--------------------------+-------------------------------------------------
  Reporter:  gisboa       |       Owner:  grass-dev@…              
      Type:  defect       |      Status:  reopened                 
  Priority:  major        |   Milestone:  6.4.0                    
 Component:  Vector       |     Version:  6.4.2                    
Resolution:               |    Keywords:  vector import clean build
  Platform:  MSWindows 7  |         Cpu:  x86-64                   
--------------------------+-------------------------------------------------
Description changed by neteler:

Old description:

> I try to use Grass with large vector files, with sizes up to 0,5GB, this
> usually results in errors during import, or if not during import, then at
> least while cleaning them. Both on Mac (Leopard intel) and Linux
> (openSuse 11 (32 and 64bit). It doesn't seem to be a problem with the
> data, as I can process all data when splitting the import, and handle the
> various slices separately.
>
> Files are too large to attach...
>
> Wouter
>
> Mac OSX
> ======
> GRASS 6.4.0RC3 (nl-rdn):/data/grassdb/nl-rdn > v.clean --o
> input=top_top10vec_gebouwen output=top_top10vec_geb2 tool=bpol
> --------------------------------------------------
> Tool: Threshold
> Break polygons: 0.000000e+00
> --------------------------------------------------
> Copying vector lines...
> Rebuilding parts of topology...
> Building topology for vector map <top_top10vec_geb2>...
> Registering primitives...
> 6041168 primitives registered
> 25000849 vertices registered
> Number of nodes: 6037532
> Number of primitives: 6041168
> Number of points: 0
> Number of lines: 0
> Number of boundaries: 3021055
> Number of centroids: 3020113
> Number of areas: -
> Number of isles: -
> --------------------------------------------------
> Tool: Break polygons
> v.clean(22861) malloc: *** mmap(size=275042304) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> ERROR: G_realloc: unable to allocate 275040036 bytes at
>        break_polygons.c:188
>
> Linux 64bit
> =======
> Rebuilding parts of topology...
> Building topology for vector map <top_top10vec_geb2>...
> Registering primitives...
> 6041168 primitives registered
> 25000849 vertices registered
> Number of nodes: 6037532
> Number of primitives: 6041168
> Number of points: 0
> Number of lines: 0
> Number of boundaries: 3021055
> Number of centroids: 3020113
> Number of areas: -
> Number of isles: -
> --------------------------------------------------
> Tool: Break polygons
> ERROR: G_realloc: unable to allocate 34400040 bytes at
> break_polygons.c:188

New description:

 I try to use Grass with large vector files, with sizes up to 0,5GB, this
 usually results in errors during import, or if not during import, then at
 least while cleaning them. Both on Mac (Leopard intel) and Linux (openSuse
 11 (32 and 64bit). It doesn't seem to be a problem with the data, as I can
 process all data when splitting the import, and handle the various slices
 separately.

 Files are too large to attach...

 Wouter


 {{{
 Mac OSX
 ======
 GRASS 6.4.0RC3 (nl-rdn):/data/grassdb/nl-rdn > v.clean --o
 input=top_top10vec_gebouwen output=top_top10vec_geb2 tool=bpol
 --------------------------------------------------
 Tool: Threshold
 Break polygons: 0.000000e+00
 --------------------------------------------------
 Copying vector lines...
 Rebuilding parts of topology...
 Building topology for vector map <top_top10vec_geb2>...
 Registering primitives...
 6041168 primitives registered
 25000849 vertices registered
 Number of nodes: 6037532
 Number of primitives: 6041168
 Number of points: 0
 Number of lines: 0
 Number of boundaries: 3021055
 Number of centroids: 3020113
 Number of areas: -
 Number of isles: -
 --------------------------------------------------
 Tool: Break polygons
 v.clean(22861) malloc: *** mmap(size=275042304) failed (error code=12)
 *** error: can't allocate region
 *** set a breakpoint in malloc_error_break to debug
 ERROR: G_realloc: unable to allocate 275040036 bytes at
        break_polygons.c:188
 }}}



 {{{
 Linux 64bit
 =======
 Rebuilding parts of topology...
 Building topology for vector map <top_top10vec_geb2>...
 Registering primitives...
 6041168 primitives registered
 25000849 vertices registered
 Number of nodes: 6037532
 Number of primitives: 6041168
 Number of points: 0
 Number of lines: 0
 Number of boundaries: 3021055
 Number of centroids: 3020113
 Number of areas: -
 Number of isles: -
 --------------------------------------------------
 Tool: Break polygons
 ERROR: G_realloc: unable to allocate 34400040 bytes at
 break_polygons.c:188
 }}}

--

-- 
Ticket URL: <https://trac.osgeo.org/grass/ticket/494#comment:6>
GRASS GIS <http://grass.osgeo.org>