[GRASS-dev] [GRASS-user] importing big vector data

[Martin Landa]

Hi,

[I moved this discussion to grass-dev ML which seems to be more appropriate]

2014-03-03 23:27 GMT+01:00 Markus Metz <markus.metz.giswork at gmail.com>:
>> -> after 110min failed with "ERROR: G_calloc: unable to allocate 101 *
>> 4 bytes of memory at allocation.c:81" (when Finding centroids for OGR
>> layer <buildings>...), memory peak 3.6GB
>
> This works for me on a desktop machine with quadcore AMD Phenom @ 4GHz
> and 16 GB RAM with Scientific Linux. Memory peak was 6.3 GB and it
> took 1 hour, but that included snapping which costs time and memory
> (my first attempt was also successful but I didn't measure processing
> time), and I processed at the same time a time series of climate data
> in a different GRASS session. There are a number of topological errors
> in the shapefile, amongst others there are 6619 overlapping areas
> which are too large to be removed by snapping and might need manual
> cleaning.
>
> The memory allocation errors you experienced are strange, they should
> not happen on a system with 48 GB RAM. Can you hack v.in.ogr and make
> it try to allocate e.g. 8 GB of memory right at the beginning? If that
> fails, you could check how much RAM is free on your system. Disk space
> can also be a limiting factor but seems unlikely in your case.

right, this strange, I used code bellow

    int i;
    int base = 1e6;
    char *buf = NULL;
    for(i = 1 ; i <= 8000; i++) {
      G_debug(0, "%d -> %ld", i, i * base);
      buf = G_realloc(buf, i * base);
      if (i % 100 == 0)
        sleep(1);
    }

it fails when trying to allocate about 2GB,

D0/0: 2047 -> 2047000000
D0/0: 2048 -> 2048000000
Current region rows: 3, cols: 7
ERROR: G_realloc: unable to allocate 2048000000 bytes of memory at main.c:101

There is memory available and no limits per process:

$ free -m
             total       used       free     shared    buffers     cached
Mem:         48395       4587      43808          0       3272        361
-/+ buffers/cache:        953      47441
Swap:        47682          0      47682

Trying to investigate more. In any case thanks for testing... Martin