[GRASS-dev] [GRASS-user] importing big vector data

[Glynn Clements]

Martin Landa wrote:

> right, this strange, I used code bellow
> 
>     int i;
>     int base = 1e6;
>     char *buf = NULL;
>     for(i = 1 ; i <= 8000; i++) {
>       G_debug(0, "%d -> %ld", i, i * base);
>       buf = G_realloc(buf, i * base);
>       if (i % 100 == 0)
>         sleep(1);
>     }
> 
> it fails when trying to allocate about 2GB,

If any part of the code leaks memory, this will fail due to heap
fragmentation.

Each allocation is slightly larger than the previous one. If something
allocates memory between the reallocations of the main block, you'll
end up with the heap looking like:

	[1Mb free][used][2Mb free][used][3Mb free][used]...

The end result will be 1+2+3+...+7999 Mb = ~32 terabytes of "free"
memory, all of it in blocks too small to be used for the next
allocation.

If it fails at i==2048, the figure would be 2 terabytes.

Does removing the G_debug call have an effect?

-- 
Glynn Clements <glynn at gclements.plus.com>