[GRASS-dev] [GRASS GIS] #2295: import of large shapefile in GRASS-db crashes 6.4.3 + 7.0 beta2

[GRASS GIS]

#2295: import of large shapefile in GRASS-db crashes 6.4.3 + 7.0 beta2
-------------------------+--------------------------------------------------
 Reporter:  janhuizinga  |       Owner:  grass-dev@…              
     Type:  defect       |      Status:  new                      
 Priority:  major        |   Milestone:  6.4.4                    
Component:  LibVector    |     Version:  6.4.3                    
 Keywords:  v.in.ogr     |    Platform:  MSWindows 7              
      Cpu:  x86-64       |  
-------------------------+--------------------------------------------------
Changes (by neteler):

  * keywords:  => v.in.ogr
  * component:  Default => LibVector


Old description:

> OS: Windows 7
> GRASS versions: 6.4 + 7.0 beta2
>
> I'm trying to import a shapefile containing 9,354,125 point of type 2.5D
> in a GRASS-db. Size of this shapefile: 328MB for shp-file + 301MB for
> dbf-file.
> If I try to import it WITH creating an attribute table it only loads half
> of the points and then exits with an error message (message attached
> below). Size of coor-file in GRASS is then 173MB.
>
> If I try to import the shapefile WITHOUT creating an attribute table it
> fully converts the points, but reports an memory error during
> "Registering primitives ..." Size of coor-file in GRASS is then 337MB
> (message attached below)
>
> Error messages in command console below:
>
> ++++++++++++++++++++++
> error 1:
> ++++++++++++++++++++++
> (Fri May 16 13:20:20 2014)
> v.in.ogr dsn=D:\Users\HKV2\PR2820_Mego\JanH\Gebouwen3\adreslocatie.shp
> layer=adreslocatie output=adreslocatie
> Check if OGR layer <adreslocatie> contains polygons...
> Importing 9354125 features (OGR layer <adreslocatie>)...
> WARNING: System not initialized. Programmer forgot to call G_gisinit().
> dbmi: Protocol error
> dbmi: Protocol error
> dbmi: Protocol error
> ERROR: Cannot insert new row: insert into adreslocatie values ( 4802001,
> '0363200000223289', '1011LH', 3, 300 )
> (Fri May 16 13:28:58 2014) Command finished (8 min 38 sec)
>
> ++++++++++++++++++++++
> error 2:
> ++++++++++++++++++++++
>
> (Fri May 16 14:58:19 2014)
> v.in.ogr dsn=D:\Users\HKV2\PR2820_Mego\JanH\Gebouwen3\adreslocatie.shp
> layer=adreslocatie output=adreslocatie --overwrite -t
> Check if OGR layer <adreslocatie> contains polygons...
> WARNING: Vector map <adreslocatie> already exists and will be overwritten
> WARNING: Table <adreslocatie> linked to vector map <adreslocatie> does
> not exist
> Importing 9354125 features (OGR layer <adreslocatie>)...
> -----------------------------------------------------
> Building topology for vector map <adreslocatie at calc>...
> Registering primitives...
> Current region rows: 516, cols: 790
> ERROR: G_realloc: unable to allocate 81480000 bytes of memory at
> cindex.c:113
> (Fri May 16 15:08:04 2014) Command finished (9 min 44 sec)

New description:

 OS: Windows 7
 GRASS versions: 6.4 + 7.0 beta2

 I'm trying to import a shapefile containing 9,354,125 point of type 2.5D
 in a GRASS-db. Size of this shapefile: 328MB for shp-file + 301MB for dbf-
 file.
 If I try to import it WITH creating an attribute table it only loads half
 of the points and then exits with an error message (message attached
 below). Size of coor-file in GRASS is then 173MB.

 If I try to import the shapefile WITHOUT creating an attribute table it
 fully converts the points, but reports an memory error during "Registering
 primitives ..." Size of coor-file in GRASS is then 337MB (message attached
 below)

 Error messages in command console below:

 {{{
 ++++++++++++++++++++++
 error 1:
 ++++++++++++++++++++++
 (Fri May 16 13:20:20 2014)
 v.in.ogr dsn=D:\Users\HKV2\PR2820_Mego\JanH\Gebouwen3\adreslocatie.shp
 layer=adreslocatie output=adreslocatie
 Check if OGR layer <adreslocatie> contains polygons...
 Importing 9354125 features (OGR layer <adreslocatie>)...
 WARNING: System not initialized. Programmer forgot to call G_gisinit().
 dbmi: Protocol error
 dbmi: Protocol error
 dbmi: Protocol error
 ERROR: Cannot insert new row: insert into adreslocatie values ( 4802001,
 '0363200000223289', '1011LH', 3, 300 )
 (Fri May 16 13:28:58 2014) Command finished (8 min 38 sec)
 }}}


 {{{

 ++++++++++++++++++++++
 error 2:
 ++++++++++++++++++++++

 (Fri May 16 14:58:19 2014)
 v.in.ogr dsn=D:\Users\HKV2\PR2820_Mego\JanH\Gebouwen3\adreslocatie.shp
 layer=adreslocatie output=adreslocatie --overwrite -t
 Check if OGR layer <adreslocatie> contains polygons...
 WARNING: Vector map <adreslocatie> already exists and will be overwritten
 WARNING: Table <adreslocatie> linked to vector map <adreslocatie> does not
 exist
 Importing 9354125 features (OGR layer <adreslocatie>)...
 -----------------------------------------------------
 Building topology for vector map <adreslocatie at calc>...
 Registering primitives...
 Current region rows: 516, cols: 790
 ERROR: G_realloc: unable to allocate 81480000 bytes of memory at
 cindex.c:113
 (Fri May 16 15:08:04 2014) Command finished (9 min 44 sec)
 }}}

--

Comment:

 Did you use GRASS 6 in both cases or does GRASS 7 generate the same
 errors?

-- 
Ticket URL: <http://trac.osgeo.org/grass/ticket/2295#comment:2>
GRASS GIS <http://grass.osgeo.org>