[gdal-dev] In Memory Raster - allocation run out of memory onbig sizes.

[mchapman at texelinc.com]

Daniele,

Are you using gdal api through jni or are you calling one of gdals exe programs?  Java limits the total memory for in-process dll's to whatever the xmx is set to minus whatever the jre needs.  I have dealt with this using java before.  You can either increase the xmx setting or call gdal from outside the java runtime, like a c++ exe.  In any case, you will always be limited to the amount of memory on your machine.  
P.S.  A good driver will use memory mapped file io (file channels in java NIO) for reading and writing which gives you a max file size of 1.8 Exobytes on windows and will consume minimal memory during execution...like a couple of MB. Unfortunately, most drivers don't use file channels. :)

Martin

Sent via BlackBerry by AT&T

-----Original Message-----
From: Frank Warmerdam <warmerdam at pobox.com>

Date: Fri, 04 Jan 2008 11:09:19 
To:Daniele Romagnoli <dany.geotools at gmail.com>
Cc:gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] In Memory Raster - allocation run out of memory on
	big sizes.


Daniele Romagnoli wrote:
> Hi list,
> I'm trying to build a very big In Memory Raster Dataset (~1 GB) using 
> SWIG's Java Bindings on my Notebook (with Windows Vista, with 2GB ram)
> 
> When calling the java method memoryDriver.create(...) the following 
> error appears "ERROR 2: Unable to create band arrays ... out of memory" 
> which is thrown when the underlying calloc method can't successfully 
> allocate memory.
> I have performed some tests and tunings with the following conclusions:
> - reducing the JVM Heap Size (using -Xmx command) allows to allocate a 
> bigger native memory portion (using -Xmx1024M allows me to allocate only 
> <550 MB memory while using -Xmx16M allows me to allocate up to ~820MB) 
> [Java Heap space and "Native" Heap space are managed separately]
> - increasing the virtual memory, the result is the same (out of memory).
> - killing all unuseful processes -> freeing more physical memory (~1.2 
> GB), the result is the same (out of memory))
> Any setting allows me to allocate a max of ~840000 KBytes.
> 
> Anyway, why having almost 1.2 GB free physical memory and a 3GB virtual 
> memory setting (with ~2GB free) allows me only to create always ~840000 
> KBytes?
> Does anyone know how memory is managed, how the calloc method operates 
> and whether an OS limitation exist?
> Thank you very much.

Daniele,

I have no reason to believe this is a GDAL specific problem.  I presume you
are running 32bit Windows Vista?  If so, I believe any one process can have
at most 2GB of heap space which in your case would presumably be split
between the JVM and non Java memory in the same process.

I think you could allocate larger in memory rasters if you were running Win64
which has a very large potential virtual memory size per process.

PS. I started out this reply to tell you to check the ulimit command output
which reports per-process memory limits on linux before I realized you were
using windows.  I don't know if there is any equivelent thing on windows.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev