[gdal-dev] In Memory Raster - allocation run out of memory on big sizes.

[Frank Warmerdam]

Daniele Romagnoli wrote:
> Hi list,
> I'm trying to build a very big In Memory Raster Dataset (~1 GB) using 
> SWIG's Java Bindings on my Notebook (with Windows Vista, with 2GB ram)
> 
> When calling the java method memoryDriver.create(...) the following 
> error appears "ERROR 2: Unable to create band arrays ... out of memory" 
> which is thrown when the underlying calloc method can't successfully 
> allocate memory.
> I have performed some tests and tunings with the following conclusions:
> - reducing the JVM Heap Size (using -Xmx command) allows to allocate a 
> bigger native memory portion (using -Xmx1024M allows me to allocate only 
> <550 MB memory while using -Xmx16M allows me to allocate up to ~820MB) 
> [Java Heap space and "Native" Heap space are managed separately]
> - increasing the virtual memory, the result is the same (out of memory).
> - killing all unuseful processes -> freeing more physical memory (~1.2 
> GB), the result is the same (out of memory))
> Any setting allows me to allocate a max of ~840000 KBytes.
> 
> Anyway, why having almost 1.2 GB free physical memory and a 3GB virtual 
> memory setting (with ~2GB free) allows me only to create always ~840000 
> KBytes?
> Does anyone know how memory is managed, how the calloc method operates 
> and whether an OS limitation exist?
> Thank you very much.

Daniele,

I have no reason to believe this is a GDAL specific problem.  I presume you
are running 32bit Windows Vista?  If so, I believe any one process can have
at most 2GB of heap space which in your case would presumably be split
between the JVM and non Java memory in the same process.

I think you could allocate larger in memory rasters if you were running Win64
which has a very large potential virtual memory size per process.

PS. I started out this reply to tell you to check the ulimit command output
which reports per-process memory limits on linux before I realized you were
using windows.  I don't know if there is any equivelent thing on windows.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org