[GRASS-dev] [GRASS-user] importing big vector data

[Blumentrath, Stefan]

Hi,

Just to add a side-note: I had no problems with importing the entire Norwegian N50 dataset in GRASS 7 (r58048 on Ubuntu Server 12.04 LTS with 48GM RAM). The N50 dataset contains 2909696 areas. And also v.overlay worked very well...

Cheers
Stefan

-----Original Message-----
From: grass-dev-bounces at lists.osgeo.org [mailto:grass-dev-bounces at lists.osgeo.org] On Behalf Of Glynn Clements
Sent: 6. mars 2014 10:01
To: Martin Landa
Cc: GRASS developers list; Markus Metz
Subject: Re: [GRASS-dev] [GRASS-user] importing big vector data


Martin Landa wrote:

> right, this strange, I used code bellow
> 
>     int i;
>     int base = 1e6;
>     char *buf = NULL;
>     for(i = 1 ; i <= 8000; i++) {
>       G_debug(0, "%d -> %ld", i, i * base);
>       buf = G_realloc(buf, i * base);
>       if (i % 100 == 0)
>         sleep(1);
>     }
> 
> it fails when trying to allocate about 2GB,

If any part of the code leaks memory, this will fail due to heap fragmentation.

Each allocation is slightly larger than the previous one. If something allocates memory between the reallocations of the main block, you'll end up with the heap looking like:

	[1Mb free][used][2Mb free][used][3Mb free][used]...

The end result will be 1+2+3+...+7999 Mb = ~32 terabytes of "free"
memory, all of it in blocks too small to be used for the next allocation.

If it fails at i==2048, the figure would be 2 terabytes.

Does removing the G_debug call have an effect?

--
Glynn Clements <glynn at gclements.plus.com> _______________________________________________
grass-dev mailing list
grass-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev