[GRASS-user] Memory problems with v.in.ogr

[Pierre Roudier]

HI,

I tried again with the koordinates shapefile. I did succeed that time
- corrupted shapefile I suppose.

Thanks for the help,

Pierre

2010/11/30 Markus Metz <markus.metz.giswork at googlemail.com>:
> Pierre Roudier wrote:
>> Dear all,
>>
>> I tried the very same thing on the stable version of GRASS (6.4 RC7 -
>> which AFAIK is the stable 6.4), on the very same machine.
>
> The stable 6.4 version is 6.4.0, the recommended 6.4 version is 6.4.1
>
>> Unfortunately, I still an error. On the "Break boundaries" state
>> during the imprt, it is stuck for a while at 4gb RAM used. Then for
>> some reason the memory consumption becomes much bigger (maximum
>> actually - 6.2GB+2Gb swap) and the process gets killed.
>>
> Weird. I assume you want to import the Land Cover DataBase version 2
> for the Northern Island of New Zealand in the New Zealand Map Grid
> projection. I have imported the Land Cover DataBase version 2 for all
> of New Zealand in the New Zealand Transverse Mercator projection with
> more than twice as many polygons as in the Northern Island only, and
> memory peak was at about 4GB. Judging by the cleaning done by v.in.ogr
> --v, the shapefile I imported was clean. For not so clean shapefiles
> where adjacent polygons are not 100% adjacent but are slightly
> overlapping or leave small gaps in between them, memory consumption
> can go up considerably.
>
> I have two suggestions:
> 1. Use the snapping option of v.in.ogr with snap=1 and have verbose
> output with --v. Considering that the LCDB2 was created from Landsat
> ETM7+ imagery, the spatial detail can not be finer than 28.5m and you
> may be able to increase the snapping threshold to values larger than 1
> without snapping away polygons.
>
> 2. Use the lcdb2 version I used, available at koordinates.com [1].
> Optionally set the region first to the Northern Island and then import
> with v.in.ogr -r
>
>> Any pointers would be much appreciated. Otherwise, maybe I should open
>> a bug for that?
>>
> I don't think this is a bug. Memory consumption can get quite high
> when processing larger vectors with topology in grass. I know which
> are the memory-hungry components and want to reduce memory
> consumption, but that will happen only in grass 7, and most probably
> not this year.
>
> Markus M
>
> [1] http://koordinates.com/layer/1072-land-cover-database-version-2-lcdb2/
>
>>
>> 2010/11/28 Pierre Roudier <pierre.roudier at gmail.com>:
>>> Thanks for your answers. Yes, this is a 64bits OS.
>>>
>>> 2010/11/27 Markus Metz <markus.metz.giswork at googlemail.com>:
>>>> On Fri, Nov 26, 2010 at 3:14 AM, Pierre Roudier
>>>> <pierre.roudier at gmail.com> wrote:
>>>>> Dear list,
>>>>>
>>>>> I am trying to load a pretty big shapefile on GRASS (6Mb +, 211621
>>>>> features)  using v.in.ogr, but each attempt fails at the 'break
>>>>> boundaries'' stage (at 98%!):
>>>>>
>>>>> GRASS 7.0.svn (NZTM2000):~ > v.in.ogr
>>>>> dsn=~/Documents/DATA/LCDB2/ni_nzmg.shp out=ni_lcdb2 --o
>>>>> Projection of input dataset and current location appear to match
>>>>> Layer: ni_nzmg
>>>>> Counting polygons for 211621 features...
>>>>> Importing map 211621 features...
>>>>>  100%
>>>>> -----------------------------------------------------
>>>>> Building topology for vector map <ni_lcdb2_tmp at ni>...
>>>>> Registering primitives...
>>>>> 334611 primitives registered
>>>>> 18459167 vertices registered
>>>>> Number of nodes: 211681
>>>>> Number of primitives: 334611
>>>>> Number of points: 0
>>>>> Number of lines: 0
>>>>> Number of boundaries: 334611
>>>>> Number of centroids: 0
>>>>> Number of areas: -
>>>>> Number of isles: -
>>>>> -----------------------------------------------------
>>>>> WARNING: Cleaning polygons, result is not guaranteed!
>>>>> -----------------------------------------------------
>>>>> Break polygons:
>>>>>  100%
>>>>>  100%
>>>>> -----------------------------------------------------
>>>>> Remove duplicates:
>>>>>  100%
>>>>> -----------------------------------------------------
>>>>> Break boundaries:
>>>>> ERROR: G_calloc: unable to allocate 50 * 8 bytes of memory at
>>>>>       allocation.c:82
>>>>>
>>>>> The shp loads without any problems on QGIS.
>>>>>
>>>>
>>>>> OS: Opensuse 11.3, with 7Gb RAM.
>>>> Is this a 64 bit OS?
>>>>
>>>> I have previously successfully imported larger shapefiles, more than
>>>> double in size: >400,000 areas, >40 million vertices, peak memory
>>>> consumption was close to 4GB. The import of your shapefile would
>>>> approximately require about 2GB of memory, which should not be a
>>>> problem with 7GB available, as long as it's all 64 bit and not 32 bit.
>>>>
>>>> Markus M
>>>>
>>>
>>
>