[gdal-dev] In Memory Raster - allocation run out of memory onbig sizes.

[Daniele Romagnoli]

Hi Martin,
I'm using GDAL through JNI (by means of the SWIG's generated java bindings).
However, I guess that increasing the xmx, "steals" memory which could be
used by native items.

Regards,
Daniele.

On Jan 4, 2008 5:32 PM, <mchapman at texelinc.com> wrote:

> Daniele,
>
> Are you using gdal api through jni or are you calling one of gdals exe
> programs?  Java limits the total memory for in-process dll's to whatever the
> xmx is set to minus whatever the jre needs.  I have dealt with this using
> java before.  You can either increase the xmx setting or call gdal from
> outside the java runtime, like a c++ exe.  In any case, you will always be
> limited to the amount of memory on your machine.
> P.S.  A good driver will use memory mapped file io (file channels in java
> NIO) for reading and writing which gives you a max file size of 1.8Exobytes on windows and will consume minimal memory during execution...like
> a couple of MB. Unfortunately, most drivers don't use file channels. :)
>
> Martin
>
> Sent via BlackBerry by AT&T
>
> -----Original Message-----
> From: Frank Warmerdam <warmerdam at pobox.com>
>
> Date: Fri, 04 Jan 2008 11:09:19
> To:Daniele Romagnoli <dany.geotools at gmail.com>
> Cc:gdal-dev at lists.osgeo.org
> Subject: Re: [gdal-dev] In Memory Raster - allocation run out of memory on
>        big sizes.
>
>
> Daniele Romagnoli wrote:
> > Hi list,
> > I'm trying to build a very big In Memory Raster Dataset (~1 GB) using
> > SWIG's Java Bindings on my Notebook (with Windows Vista, with 2GB ram)
> >
> > When calling the java method memoryDriver.create(...) the following
> > error appears "ERROR 2: Unable to create band arrays ... out of memory"
> > which is thrown when the underlying calloc method can't successfully
> > allocate memory.
> > I have performed some tests and tunings with the following conclusions:
> > - reducing the JVM Heap Size (using -Xmx command) allows to allocate a
> > bigger native memory portion (using -Xmx1024M allows me to allocate only
> > <550 MB memory while using -Xmx16M allows me to allocate up to ~820MB)
> > [Java Heap space and "Native" Heap space are managed separately]
> > - increasing the virtual memory, the result is the same (out of memory).
> > - killing all unuseful processes -> freeing more physical memory (~1.2
> > GB), the result is the same (out of memory))
> > Any setting allows me to allocate a max of ~840000 KBytes.
> >
> > Anyway, why having almost 1.2 GB free physical memory and a 3GB virtual
> > memory setting (with ~2GB free) allows me only to create always ~840000
> > KBytes?
> > Does anyone know how memory is managed, how the calloc method operates
> > and whether an OS limitation exist?
> > Thank you very much.
>
> Daniele,
>
> I have no reason to believe this is a GDAL specific problem.  I presume
> you
> are running 32bit Windows Vista?  If so, I believe any one process can
> have
> at most 2GB of heap space which in your case would presumably be split
> between the JVM and non Java memory in the same process.
>
> I think you could allocate larger in memory rasters if you were running
> Win64
> which has a very large potential virtual memory size per process.
>
> PS. I started out this reply to tell you to check the ulimit command
> output
> which reports per-process memory limits on linux before I realized you
> were
> using windows.  I don't know if there is any equivelent thing on windows.
>
> Best regards,
> --
>
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam<http://pobox.com/%7Ewarmerdam>
> and watch the world go round - Rush    | President OSGeo, http://osgeo.org
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
-------------------------------------------------------
Eng. Daniele Romagnoli
Software Engineer

GeoSolutions S.A.S.
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax:     +39 0584983027
mob:   +39 328 0559267


http://www.geo-solutions.it

-------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20080104/b681acc5/attachment.html