[gdal-dev] In Memory Raster - allocation run out of memory onbig sizes.

[Daniele Romagnoli]

Sorry Martin,
I will rectify the statement in my previous reply: "I guess that increasing
the xmx, "steals" memory which could be used by native items."
I'm saying that I have noticed this:
when increasing xmx settings, the out of memory errors in the native method
occurs with a smaller requested amount of data with respect to the case of a
lower xmx set.
As an instance,  Xmx1024 allows the native method to allocate only <550 MB
memory while Xmx16M allows to allocate up to ~820MB.

Thanks for your help.

Regards,
Daniele


On Jan 4, 2008 6:05 PM, Daniele Romagnoli <dany.geotools at gmail.com> wrote:

> Hi Martin,
> I'm using GDAL through JNI (by means of the SWIG's generated java
> bindings).
> However, I guess that increasing the xmx, "steals" memory which could be
> used by native items.
>
> Regards,
> Daniele.
>
>
> On Jan 4, 2008 5:32 PM, <mchapman at texelinc.com> wrote:
>
> > Daniele,
> >
> > Are you using gdal api through jni or are you calling one of gdals exe
> > programs?  Java limits the total memory for in-process dll's to whatever the
> > xmx is set to minus whatever the jre needs.  I have dealt with this using
> > java before.  You can either increase the xmx setting or call gdal from
> > outside the java runtime, like a c++ exe.  In any case, you will always be
> > limited to the amount of memory on your machine.
> > P.S.  A good driver will use memory mapped file io (file channels in
> > java NIO) for reading and writing which gives you a max file size of 1.8Exobytes on windows and will consume minimal memory during execution...like
> > a couple of MB. Unfortunately, most drivers don't use file channels. :)
> >
> > Martin
> >
> > Sent via BlackBerry by AT&T
> >
> > -----Original Message-----
> > From: Frank Warmerdam <warmerdam at pobox.com>
> >
> > Date: Fri, 04 Jan 2008 11:09:19
> > To:Daniele Romagnoli <dany.geotools at gmail.com>
> > Cc:gdal-dev at lists.osgeo.org
> > Subject: Re: [gdal-dev] In Memory Raster - allocation run out of memory
> > on
> >        big sizes.
> >
> >
> > Daniele Romagnoli wrote:
> > > Hi list,
> > > I'm trying to build a very big In Memory Raster Dataset (~1 GB) using
> > > SWIG's Java Bindings on my Notebook (with Windows Vista, with 2GB ram)
> >
> > >
> > > When calling the java method memoryDriver.create(...) the following
> > > error appears "ERROR 2: Unable to create band arrays ... out of
> > memory"
> > > which is thrown when the underlying calloc method can't successfully
> > > allocate memory.
> > > I have performed some tests and tunings with the following
> > conclusions:
> > > - reducing the JVM Heap Size (using -Xmx command) allows to allocate a
> > > bigger native memory portion (using -Xmx1024M allows me to allocate
> > only
> > > <550 MB memory while using -Xmx16M allows me to allocate up to ~820MB)
> > > [Java Heap space and "Native" Heap space are managed separately]
> > > - increasing the virtual memory, the result is the same (out of
> > memory).
> > > - killing all unuseful processes -> freeing more physical memory (~1.2
> > > GB), the result is the same (out of memory))
> > > Any setting allows me to allocate a max of ~840000 KBytes.
> > >
> > > Anyway, why having almost 1.2 GB free physical memory and a 3GB
> > virtual
> > > memory setting (with ~2GB free) allows me only to create always
> > ~840000
> > > KBytes?
> > > Does anyone know how memory is managed, how the calloc method operates
> > > and whether an OS limitation exist?
> > > Thank you very much.
> >
> > Daniele,
> >
> > I have no reason to believe this is a GDAL specific problem.  I presume
> > you
> > are running 32bit Windows Vista?  If so, I believe any one process can
> > have
> > at most 2GB of heap space which in your case would presumably be split
> > between the JVM and non Java memory in the same process.
> >
> > I think you could allocate larger in memory rasters if you were running
> > Win64
> > which has a very large potential virtual memory size per process.
> >
> > PS. I started out this reply to tell you to check the ulimit command
> > output
> > which reports per-process memory limits on linux before I realized you
> > were
> > using windows.  I don't know if there is any equivelent thing on
> > windows.
> >
> > Best regards,
> > --
> >
> > ---------------------------------------+--------------------------------------
> > I set the clouds in motion - turn up   | Frank Warmerdam,
> > warmerdam at pobox.com
> > light and sound - activate the windows | http://pobox.com/~warmerdam<http://pobox.com/%7Ewarmerdam>
> > and watch the world go round - Rush    | President OSGeo,
> > http://osgeo.org
> >
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
> >
>
>
>
> --
> -------------------------------------------------------
>
> Eng. Daniele Romagnoli
> Software Engineer
>
> GeoSolutions S.A.S.
> Via Carignoni 51
> 55041 Camaiore (LU)
> Italy
>
> phone: +39 0584983027
> fax:     +39 0584983027
> mob:   +39 328 0559267
>
>
> http://www.geo-solutions.it
>
> -------------------------------------------------------
>



-- 
-------------------------------------------------------
Eng. Daniele Romagnoli
Software Engineer

GeoSolutions S.A.S.
Via Carignoni 51
55041 Camaiore (LU)
Italy

phone: +39 0584983027
fax:     +39 0584983027
mob:   +39 328 0559267


http://www.geo-solutions.it

-------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/gdal-dev/attachments/20080104/f76b7ff5/attachment-0001.html