[GRASS-user] Memory problems with v.in.ogr

[Markus Metz]

Pierre Roudier wrote:
> Dear all,
>
> I tried the very same thing on the stable version of GRASS (6.4 RC7 -
> which AFAIK is the stable 6.4), on the very same machine.

The stable 6.4 version is 6.4.0, the recommended 6.4 version is 6.4.1

> Unfortunately, I still an error. On the "Break boundaries" state
> during the imprt, it is stuck for a while at 4gb RAM used. Then for
> some reason the memory consumption becomes much bigger (maximum
> actually - 6.2GB+2Gb swap) and the process gets killed.
>
Weird. I assume you want to import the Land Cover DataBase version 2
for the Northern Island of New Zealand in the New Zealand Map Grid
projection. I have imported the Land Cover DataBase version 2 for all
of New Zealand in the New Zealand Transverse Mercator projection with
more than twice as many polygons as in the Northern Island only, and
memory peak was at about 4GB. Judging by the cleaning done by v.in.ogr
--v, the shapefile I imported was clean. For not so clean shapefiles
where adjacent polygons are not 100% adjacent but are slightly
overlapping or leave small gaps in between them, memory consumption
can go up considerably.

I have two suggestions:
1. Use the snapping option of v.in.ogr with snap=1 and have verbose
output with --v. Considering that the LCDB2 was created from Landsat
ETM7+ imagery, the spatial detail can not be finer than 28.5m and you
may be able to increase the snapping threshold to values larger than 1
without snapping away polygons.

2. Use the lcdb2 version I used, available at koordinates.com [1].
Optionally set the region first to the Northern Island and then import
with v.in.ogr -r

> Any pointers would be much appreciated. Otherwise, maybe I should open
> a bug for that?
>
I don't think this is a bug. Memory consumption can get quite high
when processing larger vectors with topology in grass. I know which
are the memory-hungry components and want to reduce memory
consumption, but that will happen only in grass 7, and most probably
not this year.

Markus M

[1] http://koordinates.com/layer/1072-land-cover-database-version-2-lcdb2/

>
> 2010/11/28 Pierre Roudier <pierre.roudier at gmail.com>:
>> Thanks for your answers. Yes, this is a 64bits OS.
>>
>> 2010/11/27 Markus Metz <markus.metz.giswork at googlemail.com>:
>>> On Fri, Nov 26, 2010 at 3:14 AM, Pierre Roudier
>>> <pierre.roudier at gmail.com> wrote:
>>>> Dear list,
>>>>
>>>> I am trying to load a pretty big shapefile on GRASS (6Mb +, 211621
>>>> features)  using v.in.ogr, but each attempt fails at the 'break
>>>> boundaries'' stage (at 98%!):
>>>>
>>>> GRASS 7.0.svn (NZTM2000):~ > v.in.ogr
>>>> dsn=~/Documents/DATA/LCDB2/ni_nzmg.shp out=ni_lcdb2 --o
>>>> Projection of input dataset and current location appear to match
>>>> Layer: ni_nzmg
>>>> Counting polygons for 211621 features...
>>>> Importing map 211621 features...
>>>>  100%
>>>> -----------------------------------------------------
>>>> Building topology for vector map <ni_lcdb2_tmp at ni>...
>>>> Registering primitives...
>>>> 334611 primitives registered
>>>> 18459167 vertices registered
>>>> Number of nodes: 211681
>>>> Number of primitives: 334611
>>>> Number of points: 0
>>>> Number of lines: 0
>>>> Number of boundaries: 334611
>>>> Number of centroids: 0
>>>> Number of areas: -
>>>> Number of isles: -
>>>> -----------------------------------------------------
>>>> WARNING: Cleaning polygons, result is not guaranteed!
>>>> -----------------------------------------------------
>>>> Break polygons:
>>>>  100%
>>>>  100%
>>>> -----------------------------------------------------
>>>> Remove duplicates:
>>>>  100%
>>>> -----------------------------------------------------
>>>> Break boundaries:
>>>> ERROR: G_calloc: unable to allocate 50 * 8 bytes of memory at
>>>>       allocation.c:82
>>>>
>>>> The shp loads without any problems on QGIS.
>>>>
>>>
>>>> OS: Opensuse 11.3, with 7Gb RAM.
>>> Is this a 64 bit OS?
>>>
>>> I have previously successfully imported larger shapefiles, more than
>>> double in size: >400,000 areas, >40 million vertices, peak memory
>>> consumption was close to 4GB. The import of your shapefile would
>>> approximately require about 2GB of memory, which should not be a
>>> problem with 7GB available, as long as it's all 64 bit and not 32 bit.
>>>
>>> Markus M
>>>
>>
>